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EXECUTIVE  SUMMARY 


Control  applications  involving  multiple  Unmanned  Aerial  Vehicles  (UAVs)  will  require  the 
operator  to  switeh  attention  between  UAVs,  each  potentially  involving  very  different  scenario 
environments  (terrain,  threat  environment,  mission  objeetives,  weather,  ete.)  and  task 
requirements.  Not  only  is  a  potential  for  negative  effeets  assoeiated  with  task  interruptions  and 
the  mental  effort  required  in  “eontext  aequisition”  after  the  switch,  there  also  is  a  potential  for 
negative  transfer  of  eontext  to  oecur.  Such  that,  the  speeific  information  and  tasking  involved  in 
the  previous  mission  might  delay  or  degrade  the  operator’s  ability  to  effeetively  perform  tasks  in 
a  new  mission.  A  transition  aid  that  employs  synthetie  vision  teehnology  designed  to  enhanee  an 
operator’s  situation  awareness  when  switching  between  missions  in  a  multi-UAV  control 
environment  is  now  under  evaluation.  Instead  of  diseretely  switching  from  the  eamera  view  on 
one  UAV  to  the  eamera  view  of  another,  a  transition  format  is  presented  sueh  that  the  eamera 
imagery  seamlessly  fades  into  a  synthetie  imagery  eorrelate  of  the  real  video  image.  It  then  uses 
a  “fly-out,  fly-in”  metaphor  over  several  seeonds  and  finishing  with  the  transition  baek  from 
synthetie  to  real  video  imagery  at  the  new  camera  viewpoint.  During  transition,  points  of  interest 
(threats,  landmarks,  runways,  ete.)  are  eontinuously  highlighted  with  overlaid,  geo-registered 
eomputer-generated  symbology. 

The  objeetive  of  reeent  evaluations  was  to  examine  whether  this  transition  aid  would  enhanee  a 
multi-UAV  operator’s  overall  situation  awareness  and  improve  performanee  on  a  target 
designation  task  after  switehing  UAV/eamera  views.  First,  two  pilot  studies  were  conducted  to 
help  design  the  transition  format  for  multi-UAV  applieations.  Then,  a  full  simulation  evaluation 
was  eondueted  that  manipulated  the  nature  of  the  source  mission  and  assessed  whether  the 
transition  had  any  negative  effeets  on  partieipants’  eompletion  of  tasks  in  the  new  mission,  as 
well  as  seeondary  mission-related  tasks  (e.g.,  a  eommunications  task).  The  experiment  utilized 
the  Vigilant  Spirit  multi-UAV  operator  eontrol  station  testbed  developed  by  the  US  Air  Foree 
Researeh  Laboratory.  The  station  ineluded  a  (simulated)  eamera  view  from  the  seleeted  UAV, 
thumbnail  eamera  views  from  the  other  UAVs,  Taetieal  Situation  Displays  showing  the  loeation 
of  four  UAVs  as  well  as  a  elose  in  view  of  the  selected  UAV,  and  windows  used  for  seeondary 
mission-related  tasks. 

Results  were  mixed,  but  very  informative.  Partieipants’  subjeetive  ratings  on  questionnaires 
indieated  that  they  had  more  situation  awareness  in  trials  with  the  transition  aid,  eompared  to 
trials  without  it.  However,  they  failed  to  perform  better  on  a  probe  administered  during  the 
mission  that  was  designed  to  measure  eontext-speeifie  situation  awareness.  While  the  transition 
was  not  found  to  hinder  performanee  on  seeondary  tasks,  it  also  did  not  impaet  performanee  on 
the  key  task  -  the  average  time  to  loeate/designate  targets  was  only  slightly  faster  when  the 
transition  was  utilized.  The  transition  however,  did  improve  the  target  designation  task  in  terms 
of  eamera  movement  effieieney  (aeeuraey  of  initial  eamera  movement  and  eamera  path  length). 
Several  potential  enhaneements  to  the  transition  aid  were  identified,  ranging  from  the  speed  of 
various  segments  of  the  transition  to  whether  or  not  the  operator  has  direet  control  over  transition 
parameters.  Follow-on  researeh  will  examine  the  nature  of  the  speeific  missions  involved  when 
switehing  UAV/eamera  views  as  well  as  operator  strategy. 
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INTRODUCTION 


Unmanned  Aerial  Vehicle  (UAV)  operators  must  rely  on  video  imagery  transmitted  from  one  or 
more  cameras  mounted  on  the  air  platform  for  maintaining  situation  awareness  [1].  In  many 
cases,  this  video  camera  is  mounted  on  a  gimbaled  turret  and  thus  can  be  rotated  by  a  remotely 
situated  sensor  operator  in  order  to  view  various  points  of  interest  in  the  UAV’s  surrounding 
environment.  UAV  pilots  use  this  imagery  to  verify  clear  path  for  taxi/runway  operations,  scan 
for  other  air  traffic  in  the  area,  and  identify  landmarks  and  potential  obstructions.  Sensor 
operators  use  the  imagery  to  conduct  a  wide  variety  of  intelligence,  surveillance,  and 
reconnaissance  activities  as  well  as  to  directly  support  combat  operations.  However,  video 
imagery  quality  (and  by  extension,  operator  situation  awareness)  is  often  compromised  by 
narrow  camera  field-of-view,  datalink  degradations,  poor  environmental  conditions  (e.g., 
dawn/dusk/night,  adverse  weather,  variable  clouds),  bandwidth  limitations,  and  highly  cluttered 
visual  scenes  (e.g.,  in  urban  area  or  mountainous  terrain). 

A  system  that  superimposes  computer-generated  graphics  over  real  world  imagery  can 
potentially  enhance  the  video  interpretation  and  situation  awareness  of  the  UAV  operator, 
consequently  improving  decision  making  [2].  With  this  technology,  non-stereoscopic  imagery 
from  the  UAV  camera(s)  (the  substratum  defining  the  principal  scene  being  presented  from  the 
real  world)  is  presented  on  a  monitor  along  with  computer  graphic  images  that  are  overlaid  to 
create  an  enhanced  view  [3].  Specifically,  the  spatially-relevant  information,  created  from 
databases  (e.g.,  terrain,  maps,  photo-imagery,  pre-mission  plan,  etc.)  and  updates  from  other 
sources  (via  networked  communications),  can  be  represented  as  computer  symbology  and 
simulated  imagery  and  overlaid  conformal  onto  the  dynamic  camera  video  imagery  that  is 
presented  to  operators.  Figure  1  shows  an  example  of  colored  synthetic  symbology  added  to 
simulated  UAV  gimbal  video  imagery,  with  symbology  marking  threats,  landmarks,  areas  of 
interest,  and  a  runway. 


Figure  1.  Illustration  of  Synthetic  Symbology  Added  to  UAV  Video  Imagery 
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Use  of  synthetic  vision  and  augmented  technology  is  expected  to  improve  UAV  operator 
situation  awareness  by  highlighting,  in  real  time,  key  information  elements  of  interest  directly  on 
the  camera  video.  Also,  the  system  can  include  information  that  does  not  have  a  correlate  in  the 
actual  sensor  imagery,  such  as  lethality  envelopes  of  ground-based  threats.  The  system  can 
convey  self-motion  cues  and  depth  cues  without  occluding  the  sensor  image.  It  may  also  help  an 
operator  maintain  situation  awareness  during  periods  of  video  datalink  degradation  and  poor 
visibility,  through  a  combination  of  real  and  synthetic  imagery.  At  a  maximum  setting,  the 
synthetic  vision  imagery  would  totally  replace  the  real  sensor  image,  while  other  settings  would 
specify  a  blending  of  the  two  information  sources  by  changing  the  transparency  of  the  synthetic 
vision  imagery.  Additionally,  an  augmented  display  system  can  serve  a  key  role  in  supporting 
distributive  collaborative  communication  in  a  net-centric  battlespace  environment.  In  this  case, 
the  system  is  applied  both  as  a  display  and  as  a  control,  enabling  a  net-centric  member  to  mark  a 
specific  spatially  referenced  point  of  interest  on  the  camera  display,  causing  matching 
informative  synthetic  symbology  to  appear  on  the  displays  of  other  geographically  separated 
stations  in  the  warfare  network  [4]. 

All  of  the  aforementioned  expected  benefits  will  be  of  increasing  importance  in  future 
envisioned  scenarios  that  will  require  multiple  semi-autonomous  UAVs  to  be  controlled  by  a 
single  supervisor.  Compared  to  current  UAV  systems  that  require  one  or  more  operators  to 
control  a  single  UAV,  single  operator  supervisory  control  of  multiple  UAVs  is  anticipated  to  be  a 
particularly  time-critical,  cognitively  demanding  task  [5].  Even  with  highly  autonomous  UAVs, 
operators  will  need  to  respond  to  changes  in  mission  requirements,  constantly  monitor  for 
unscheduled  changes  in  information  sources,  supervise  autonomous  services,  and  intermittently 
collaborate  and  communicate  with  others  in  the  distributed  control  network.  Tasking  will  also 
require  switching  attention  from  one  UAV/camera  view  to  another.  For  scenarios  in  which 
multiple  UAVs  are  monitoring  the  same  object/scene,  camera  view  transition  aids  that  help  the 
operator  make  sense  of  how  the  different  camera  images  are  related  to  one  another  spatially 
would  be  useful  to  help  maintain  operator  situation  awareness.  Transition  aids  that  help  the 
operator  when  the  UAVs  are  monitoring  different  objects/scenes  would  also  be  useful.  In  this 
case  each  UAV  could  entail  drastically  different  scenario  conditions  (terrain,  threat  environment, 
mission  objectives,  weather,  aircraft  emergencies/malfunctions,  etc.).  Plus,  the  type  of  mission 
can  vary  [6].  In  interviews  with  UAV  controllers,  Intelligence/Surveillance/Reconnaissance 
(ISR)  missions  can  be  placed  into  two  general  categories:  1)  dynamic  missions  which  are  less 
frequent  and  include  a  rapidly  evolving  situation,  time-critical  tasks,  and  coordination  between 
multiple  assets  (e.g.,  use  of  ordinance,  tracking  a  moving  target,  and  handling  emergencies)  and 
2)  static  missions  that  are  more  common  and  generally  involve  surveillance  on  a  static  position 
for  an  extended  period  of  time. 

Dynamic  missions  are  likely  to  require  the  operator’s  full  attention.  Thus,  any  static  missions 
would  be  “on  hold”  until  the  dynamic  mission  is  completed.  It  is  probable  that  the  multi-UAV 
operator  will  witch  between  dynamic  and  static  missions,  each  potentially  involving  very 
different  scenario  environments  and  task  requirements.  Not  only  is  there  a  potential  for  the 
negative  effects  associated  with  task  interruptions  and  the  mental  effort  required  in  “context 
acquisition”  after  the  switch  [7],  there  also  is  the  potential  for  negative  transfer  of  context  to 
occur,  such  that  the  specific  information  and  tasking  involved  in  the  previous  mission  might 
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delay  or  degrade  the  operator’s  ability  to  effeetively  perform  tasks  in  the  new  mission.  For 
instanee,  if  the  operator  has  a  mental  model  of  friendly  forees  being  south  of  the  target  in  the  first 
mission,  will  the  operator  inappropriately  apply  this  mental  model  to  the  new  mission?  If  the 
eamera  is  oriented  north,  with  the  UAV  moving  in  the  same  direetion  before  the  switch,  will  it  be 
hard  for  the  operator  to  quickly  acquire  the  new  camera  orientation  and  direction  of  UAV 
movement  for  the  new  mission?  For  these  scenarios  where  the  UAVs  are  monitoring  different 
objects/scenes,  the  objective  of  the  transition  aid  would  be  to  help  the  operator  dissociate  from 
the  context/spatial  relationships  for  the  first  UAV/camera  view  and  rapidly  acquire  needed 
situation  awareness  of  the  new  UAV/camera  view. 

Regardless  of  the  application  employing  multiple  cameras,  an  operator,  when  abruptly  switched 
from  one  camera  viewpoint  to  another,  needs  time  to  acquire  situation  awareness  -  an 
understanding  of  the  elements  of  the  new  environment.  “Visual  momentum”  is  one  construct 
that  has  been  used  to  aid  the  transition  from  one  view  to  another.  Inspired  by  cinematography 
techniques  to  help  audiences  maintain  spatial  understanding  of  a  scene  across  discrete  film  cuts, 
use  of  a  continuous  transformation  between  views  to  create  visual  momentum  “supports  the 
rapid  comprehension  of  data  following  the  transition  to  a  new  display”  [8,  p.  231]. 

Previous  research  has  demonstrated  the  utility  of  visual  momentum,  showing  an  improvement  in 
task  switching  when  operators  are  provided  with  a  transition  between  two-dimensional  and  three- 
dimensional  views  of  the  same  scene  [9]  and  improvements  in  spatial  judgments  with  transitions 
between  different  perspective -rendered  views  of  the  same  scene  [10].  The  use  of  smooth 
transitions  between  two  and  three-dimensional  views  for  air  traffic  control  displays  has  also  been 
explored  [11].  A  “RealityFlythrough”  tele-reality/telepresence  system  is  introduced  [12]  to 
provide  dynamic  transitions  between  cameras  viewing  the  same  scene  in  order  to  help  the  user 
generate  an  internal  model  of  the  view.  When  the  user  is  not  viewing  real  images  generated  by 
live  cameras,  a  dynamic  fly-through  transition  is  presented  as  the  user  moves  from  one  camera 
view  to  another,  providing  useful  cues  on  the  spatial  relationships  between  the  cameras. 

Morphing  techniques  that  combine  interpolations  of  shape  and  color  have  been  used  to 
successfully  produce  compelling  transitions  between  two-dimensional  images,  as  well  as 
between  three-dimensional  views  of  either  the  same  scene  or  different  scenes  [13].  An  example 
of  how  view  morphing  can  be  applied  to  Predator  UAV  camera  images,  as  well  as  scenes  that 
contain  moving  objects,  is  available  at  [14]. 

An  Augmented  Virtual  Reality  system  designed  to  display  abstract  information  constructed  from 
multiple  sensors,  as  well  as  allowing  access  to  raw  sensor  information  such  as  video  streams,  has 
been  tested  in  three  scenarios,  each  requiring  a  different  mixture  of  both  capabilities  [15].  For 
each  camera  switching  event,  the  scene  seamlessly  fades  into  a  virtual  world  scene  that  is  in-sync 
with  the  real-life  scene.  The  user  completes  a  virtual  fly-though  before  the  scene  transitions  back 
to  the  real-life  video  stream  of  the  new  camera  view.  The  smooth  transitions  using  virtual  fly- 
though  are  designed  to  move  the  user  between  camera  views  while  maintaining  the  user’s 
relationship  with  the  objects  and  events  in  the  real  world.  The  test  results  showed  that  users  were 
able  to  maintain  a  good  spatial  sense  of  the  environment  due  to  the  smooth  transitions,  and  were 
able  to  operate  a  ground  robot  without  any  delay  after  each  camera  change. 
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To  date,  efforts  have  primarily  foeused  on  transitioning  between  ground-based  eamera  views  of 
the  same  object/seene.  Using  augmented  reality  teehnology,  the  user  is  provided  computer¬ 
generated  views  not  served  by  the  physical  cameras  to  help  the  user  retain  context  and  spatial 
relationships  with  respect  to  the  scene  when  transitioning  between  the  old  and  new  viewpoints. 
The  results  from  these  efforts  inform  the  design  of  a  transition  display  for  multi-UAV 
applications  that  involve  more  than  one  UAV  viewing  the  same  object/scene.  The  present  report 
describes  efforts  performed  by  the  Air  Force  Research  Laboratory,  71 1^’’  Human  Performance 
Wing,  Human  Effectiveness  Directorate,  Warfighter  Interface  Division’s,  Supervisory  Control 
Interfaces  Branch  (7 1 1  HPW/RHCI)  to  develop  a  transition  display  format  designed  to  help  a 
UAV  operator  transition  between  camera  views  for  applications  requiring  two  or  more  airborne 
vehicles  monitoring  different  objects/scenes.  The  goal  for  this  application  is  to  help  the  operator 
dissociate  from  the  context/spatial  relationships  associated  with  the  first  UAV/camera  view  and 
rapidly  acquire  needed  situation  awareness  of  the  new  camera  view,  reducing  the  potential  for 
negative  transfer  of  context  to  occur.  This  acquisition  of  situation  awareness  can  be  described  as 
“Getting  Into  The  Zone,”  and  hence  the  71 1  HPW/RHCI  format  under  development  is  referred  to 
as  the  “GITZ  transition.” 
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711  HPW/RHCI  ‘GITZ’  TRANSITION  DISPLAY  FORMAT 
FOR  CHANGING  CAMERA  VIEWS 


The  711  HPW/RHCI  “Get  In  The  Zone”  (GITZ)  transition  display  eoneept  is  designed  to 
enhanee  an  operator’s  situation  awareness  when  switehing  between  missions  in  a  multi-UAV 
eontrol  environment  where  eaeh  assoeiated  eamera  is  viewing  a  different  objeet/seene.  Instead 
of  diseretely  switehing  from  the  eamera  view  of  one  UAV  to  the  eamera  view  of  another,  the 
algorithms  driving  the  transition  automatieally  interpolate  to  provide  a  display  format  that 
dynamieally  ehanges  between  the  souree  and  new  eamera  views  in  a  semi-eontinuous  manner. 
This  dynamie  transition  takes  several  seeonds  and  uses  a  “fly-out,  fly-in”  metaphor  utilizing 
synthetie  vision  teehnology.  More  speeifieally,  the  transition  provides  a  three-dimensional 
perspeetive  of  synthetie  ground  imagery  from  varying  altitudes  as  the  operator  switehes  from  an 
egoeentrie  view  (determined  by  the  eamera’s  orientation/viewpoint  on  the  eurrent  UAV),  to  an 
exoeentrie  view  (a  global  view  not  tied  to  any  one  UAV;  “bird’s  eye  view”),  and  then  baek  to  an 
egoeentrie  view  (determined  by  the  eamera’s  orientation  on  the  newly  seleeted  UAV).  During 
this  transition,  points  of  interest  are  highlighted  with  overlaid,  geo-registered  eomputer-generated 
symbology. 

Figure  2  illustrates  the  key  segments  of  the  transition  display  format  (fly-out,  traverse,  and  fly- 
in).  Additionally,  it  shows  the  numerous  design  issues  to  eonsider  in  implementing  a  transition 
format.  For  the  fly-out  and  fly-in  segments,  what  path  should  the  virtual  eamera  take,  moving  at 
what  rate,  and  for  how  long?  Regarding  the  traverse  segment,  if  the  operator  was  transitioning 
between  two  eamera  views  of  the  same  target,  then  this  segment  would  be  important  to  help 
retain  eontext  and  spatial  relationships.  However,  for  the  targeted  applieation  where  the  eamera 
views  are  ehanging  from  one  geographieal  area  to  another,  showing  the  seene  between  the  two 
environments  is  not  of  interest.  Manipulation  of  the  parameters  listed  in  eaeh  oval  shown  in 
Figure  2,  ean  ehange  how  the  transition  is  pereeived  by  the  operator  and  potentially  its  utility. 
Another  researeh  question  is  the  degree  to  whieh  the  operator  should  have  eontrol  over  the 
transition  parameters  in  eaeh  segment. 

To  help  determine  optimal  GITZ  transition  parameter  settings,  we  eondueted  two  pilot  studies 
foeusing  on  one  segment  of  the  transition.  Later,  we  evaluated  a  GITZ  transition  in  a  high 
fidelity  multi-UAV  simulation.  For  this  evaluation,  both  objeetive  and  subjeetive  data  were 
reeorded  from  thirteen  partieipants  performing  multiple  tasks  while  periodieally  switehing  UAV 
missions  and  assoeiated  eamera  views.  This  report  presents  the  proeedures  and  results  from  both 
informal  pilot  studies  and  the  formal,  larger  seale  simulation  evaluation.  Reeommendations  for 
follow-on  evaluations  are  also  made. 
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“FLY-IN” 


Figure  2.  Illustration  of  Design  Issues  for  each  Segment  of  the  Transition  Format 
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PILOT  STUDY  1 


Purpose 

The  purpose  of  Pilot  Study  1  was  to  perform  a  subjeetive  assessment  of  the  fly-in  portion  of  the 
GITZ  transition,  evaluating  several  different  fly-in  eoneepts,  and  durations.  The  fly-in  portion  of 
the  transition  must  be  able  to  provide  operators  with  the  visual  eues  needed  to  rapidly  develop  a 
eognitive  map  of  spatial  points  of  interest  relative  to  the  environment  in  the  sensor  view.  By 
providing  operators  with  a  fly-in  that  ineorporates  a  bird’s  eye  view  of  the  environment,  we 
hoped  to  be  able  to  improve  situation  awareness  beyond  the  sensor’s  Field-Of-View  (FOV). 

This  type  of  situation  awareness  eould  result  in  faster,  more  effieient  responses  to  mission 
requirements  with  fewer  errors.  Another  objeetive  for  the  fly-in  is  to  have  the  transition  be 
visually  appealing  sueh  that  the  movement  and  flow  of  information  is  not  distraeting  and  does 
not  eause  negative  physieal  effeets. 


Fly-in  Development 

In  order  to  ereate  sueh  a  fly-in,  three  questions  had  be  answered:  1)  where  should  the  fly-in 
begin,  2)  what  type  of  path  should  the  fly-in  take,  and  3)  what  is  an  appropriate  duration  for  a 
fly-in?  We  determined  that  operators  should  be  able  to  see  the  UAV’s  entire  area  of  influenee  at 
the  beginning  of  the  fly-in  phase  so  they  would  have  a  perspeetive  of  the  total  environment  and 
gain  a  eontext  for  the  sensor’s  FOV.  After  trying  several  different  pitehes  for  the  fly-in  start 
point,  we  found  that  -85  degrees  from  horizontal  was  the  best  initial  piteh  for  providing  a  view  of 
the  area  surrounding  the  UAV.  This  piteh  was  optimal  beeause  it  gave  a  slight  bias  towards  the 
sensor’s  FOV  whieh  kept  it  in  the  pieture  better  during  the  fly-in  and  the  slight  tilt  ereated  a 
smoother  transition.  In  our  simulated  seenario,  the  UAV  had  an  area  of  influenee  of  12  Nautieal 
Miles  (NM)  as  the  maximum  optimal  viewing  range  for  the  sensor  was  four  NM  and  the  UAV’s 
loiter  pattern  was  a  eirele,  four  NM  in  diameter.  In  order  to  fit  this  area  into  the  Virtual  Camera 
(VC)  view  at  the  beginning  of  the  fly-in,  the  fly-in  had  to  start  at  an  altitude  of  33,000  ft  AGL 
(Above  Ground  Level)  as  the  maximum  FOV  was  48  degrees.  An  issue  that  arose  while 
eonfiguring  the  fly-in  start  point  was  the  initial  heading.  When  transitioning  from  one  UAV  to 
another,  the  heading  of  the  sensors  eould  be  different.  The  heading  would  need  to  ehange  at 
some  point  in  order  to  ereate  a  eontinuous  transition.  We  had  several  ideas  for  how  to  do  this, 
whieh  would  potentially  yield  different  headings  at  the  fly-in  start-point.  We  determined  through 
informal  testing  that  we  did  not  want  the  heading  to  ehange  during  the  fly-in  portion  of  the 
transition  as  it  would  be  too  disorienting.  Thus,  the  initial  heading  was  set  to  mateh  the  sensor’s 
heading.  Any  ehanges  in  heading  will  oeeur  during  the  traverse  or  possibly  the  fly-out  segment 
of  the  transition. 

The  end  point  for  the  fly-in  was  eontrolled  by  the  position  of  the  UAV’s  sensor.  For  the 
purposes  of  this  study,  the  UAV  was  always  fixed  at  10,000  ft  AGL  with  its  sensor  at  a  negative 
49.2  degrees  piteh  and  2.6  degrees  FOV  to  simulate  real-world  settings  for  a  taetieal  UAV 
performing  surveillanoe/reeonnaissanee. 
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With  the  start  and  end  points  in  place,  the  fly-in  path  had  to  be  determined.  The  fly-in  path  had 
to  provide  the  operator  with  a  smooth,  intuitive  descent  from  the  beginning  of  the  fly-in  to  the 
end  at  the  sensor’s  viewpoint.  Numerous  path  types  were  proposed,  of  which  four  were  chosen 
based  on  informal  testing  and  technological  limitations:  Short  Vertical  Drop  (SVD),  Long 
Vertical  Drop  (LVD),  Exponential  Sweep  (ES),  and  Einear  Sweep  (ES).  These  fly-ins  were 
divided  into  two  phases,  drop  and  zoom-in.  During  the  drop  phase,  the  VC  would  start  with  the 
UAV’s  entire  area  of  influence  in  sight  at  33,000  ft  AGE  and  descend  at  a  linear  rate  to  10,000  ft 
AGE  while  reducing  the  EOV  from  48  degrees  to  16  degrees,  decreasing  the  area  in  the  camera 
view.  Eor  SVD  and  EVD,  the  pitch  did  not  change  during  this  phase.  Whereas  for  ES,  the  pitch 
increased  to  49.2  degrees  at  a  linear  rate  and  for  ES,  the  pitch  increased  to  49.2  degrees  at  an 
exponential  rate  of  negative  0.5.  The  duration  of  the  flrst  phase  of  the  fly-in  was  varied  between 
two,  four,  and  six  seconds  to  determine  the  best  time.  Eor  the  zoom-in  phase  of  the  fly-in,  the 
altitude  remained  at  10,000  ft  AGE  and  the  EOV  decreased  at  an  exponential  rate  of  negative 
0.07  to  2.6  degrees  for  all  four  fly-ins.  The  zoom-in  phase  lasted  one  second  for  all  of  the  fly-ins 
except  EVD  which  lasted  three  seconds.  Eor  SVD  and  EVD,  the  pitch  increased  linearly  to  49.2 
degrees  during  zoom-in.  The  pitch  for  ES  and  ES  did  not  change  during  the  zoom-in  phase  of 
the  fly-in  because  it  already  matched  that  of  the  sensor.  During  the  second  zoom-in  portion  of 
the  fly-in,  the  negative  exponential  rate  of  narrowing  the  EOV  effectively  enhanced  the 
perception  of  the  Virtual  Camera  slowing  its  velocity  as  it  came  to  a  stop.  This  all  occurred  in  a 
fairly  smooth  and  continuous  manner  with  no  pauses  or  excessively  sharp  changes  in 
perspective.  By  varying  the  duration  of  the  fly-in,  we  were  able  to  create  twelve  different  fly-ins 
for  evaluation. 

The  duration  of  the  fly-in  was  of  particular  importance  as  it  had  long  enough  to  provide  operators 
with  the  necessary  visual  cues,  but  not  so  long  that  it  kept  operators  from  being  able  to  perform 
their  missions  in  a  timely  manner.  One  of  the  main  goals  of  the  GITZ  transition  is  to  provide 
operators  with  a  tool  that  can  aid  them  in  rapidly  acquiring  situation  awareness,  with  an  emphasis 
on  rapidly.  In  order  for  the  GITZ  transition  to  be  recommended,  it  must  be  able  to  improve 
situation  awareness  after  switching  without  degrading  performance  on  mission  required  tasks. 
One  of  the  key  performance  criteria  by  which  the  GITZ  transition  will  be  measured  is  the  time 
required  for  task  completion.  The  GITZ  transition  will  take  time  that  would  otherwise  be  used 
by  the  operator  to  complete  the  task.  Eor  this  reason,  the  duration  of  the  transition  cannot  exceed 
the  possible  performance  beneflts  that  could  be  realized  by  its  implementation.  As  the  GITZ 
transition  was  still  untested  at  this  point  in  time,  the  performance  benefits  were  unknown. 
Therefore,  in  this  pilot  study,  we  used  subjective  measures  of  situation  awareness  and  preference 
to  establish  a  baseline  from  which  to  work. 

Fly-out  Development 

The  fly-out  segment  of  the  GITZ  transition  would  precede  the  fly-in  segment.  Its  purpose  would 
give  operators  visual  cues,  letting  them  know  that  they  were  leaving  one  UAV  and  going  to 
another.  It  was  thought  that  this  would  allow  them  to  mentally  dissociate  themselves  from  the 
first  UAV’s  environment  and  prepare  themselves  for  the  second.  This  distinction  between 
environments  might  lead  to  fewer  errors  caused  by  negative  transfer  of  context.  The  fly-out 
portion  of  the  transition  was  not  the  focus  of  this  study,  but  one  was  included.  A  fly-out  was 
created  that  was  nearly  identical  to  the  fly-in,  but  reversed.  The  fly-out  started  at  10,000  ft  AGE 
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with  a  2.6  degree  FOV  and  negative  65  degree  piteh  and  opened  its  FOV  to  16  degrees  at  an 
exponential  rate  of  negative  0.5  over  the  eourse  of  0.5  seconds.  Then,  while  decreasing  its  pitch 
to  negative  90  degrees  at  a  linear  rate,  was  ascended  to  33,000  ft  AGL  in  1.5  seconds.  The  main 
difference  between  the  fly-out  and  the  fly-in  was  time.  Initially,  both  took  the  same  amount  of 
time,  but  we  found  that  we  needed  more  time  for  the  fly-in  phase.  The  transition  time  constraints 
required  us  to  reduce  the  duration  of  the  fly-out  in  order  to  increase  the  duration  of  the  fly-in. 

The  fly-out  duration  was  changed  to  0.5  seconds,  but  the  overwhelming  consensus  was  that  this 
was  too  fast  and  created  confusion.  The  presumed  reason  for  this  was  the  brain  still  trying  to 
process  the  new  information  that  appeared  during  the  fly-out  while  the  fly-in  started,  diverting 
attentional  resources  from  the  main  focus.  The  duration  was  incrementally  increased  to  two 
seconds,  at  which  point  the  developers  concluded  that  it  was  no  longer  a  distraction.  This  was 
deemed  an  appropriate  action  as  the  objectives  of  the  two  segments  were  different.  The 
objective  of  the  fly-out  was  to  separate  the  operator  from  the  old  environment.  In  contrast,  the 
objective  of  the  fly-in  was  to  provide  operators  with  visual  cues  which  would  allow  them  to 
develop  a  cognitive  map  of  spatial  points  of  interest  relative  to  the  new  sensor  view  in  the  new 
environment.  More  time  is  required  to  accomplish  the  objective  of  the  fly-in.  An  informal 
checkout  showed  a  preference  for  a  four  second  fly-in,  which,  given  our  five  to  seven  second 
duration  constraint,  left  one  to  three  seconds  for  the  fly-out. 

Method 

Experimental  Design 

In  order  to  evaluate  the  perceptual  impact  of  the  four  fly-in  concepts  (Short  Vertical  Drop;  SVD, 
Long  Vertical  Drop:  LVD,  Exponential  Sweep:  ES,  and  Linear  Sweep:  LS)  and  three  durations 
of  drop  time  (two,  four,  and  six  seconds),  subjective  data  were  collected  from  six  participants 
(mean  age  =  26.2  years)  using  a  within-subjects  design.  The  participants  were  asked  to  evaluate 
a  series  of  paired  fly-ins,  with  each  pair  comprised  of  two  different  fly-in  concepts  with  the  same 
drop  duration.  After  viewing  each  pair  of  fly-ins,  participants  were  asked  to  compare  the  two  on 
a  short  questionnaire.  Preceding  every  fly-in,  a  fly-out  was  presented  from  the  first  camera  view. 
Parameters  for  the  fly-out  were  not  manipulated  in  this  study. 

A  balanced  paired  comparison  design  was  used  in  which  all  possible  comparison  pairs  were 
tested  within  each  of  the  three  blocks  of  the  drop  duration  time  variable  (two,  four,  or  six 
seconds).  To  control  for  order  bias,  each  fly-in  concept  was  compared  to  every  other  fly-in 
concept  for  a  given  drop  duration  time  twice,  with  each  concept  presented  first  in  one 
comparison  and  presented  second  in  the  second  comparison.  Eor  example,  in  one  comparison 
participants  were  presented  with  SVD  fly-in  first,  followed  by  the  LVD  fly-in,  and  then  asked  to 
compare  the  two  in  the  Post-Trial  Questionnaire.  Later  in  the  experimental  session,  participants 
were  presented  with  the  same  two  fly-ins  in  the  opposite  order,  first  the  LVD  fly-in  and  then  the 
SVD  fly-in,  followed  by  the  questionnaire.  Within  each  drop  time  block,  12  comparisons  were 
made,  the  order  of  which  was  randomly  determined  with  the  following  constraints:  a)  the  two 
orders  of  any  pair  of  fly-ins  could  not  occur  consecutively  (e.g.,  SVD  -  LVD  and  then  LVD  - 
SVD)  and  b)  no  more  than  two  consecutive  comparisons  could  have  the  same  fly-in  concept 
occur  first  in  the  two  pairs. 
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The  order  of  the  three  drop  time  blocks  was  counterbalanced  across  participants  such  that  each 
drop  time  block  (two,  four,  or  six  seconds)  followed  each  other  drop  time  block  an  equal  number 
of  times  across  participants.  Participants  were  randomly  assigned  to  one  of  the  six  orders  of  the 
three  drop  time  blocks.  A  total  of  36  trials  (12  paired  comparisons  x  three  drop  time  blocks) 
were  conducted  for  each  participant  and  each  trial  lasted  approximately  two  minutes  (including 
questionnaire  time).  Total  session  time,  per  participant,  was  approximately  two  hours  (including 
15  minutes  for  training). 

Simulation  Environment 

The  study  was  performed  using  711  HPW/RHCTs  Open  Scene  Environment  (OSE)  version 
0.4.67  visualization  software  to  present  participants  with  a  manipulated  synthetic  camera  view 
that  moved  along  a  preset  path  in  an  urban  environment.  The  computer  used  for  this  experiment 
was  a  Dell  Precision  Workstation  670,  Pentium  Xeon  with  dual  3.6GHz  processors,  1GB  RAM, 
and  a  512MB  PCIe  nVidia  Quadro  FX  video  card.  Two  17”  flat  screen  LCD  monitors  were 
used;  participants  viewed  the  left  monitor,  while  the  experimenter  controlled  experimental 
conditions  with  a  keyboard  and  mouse  at  the  right  monitor  (Figure  3).  Pairs  of  fly-in  concepts 
were  shown  back  to  back  using  command-line  arguments  in  the  Fedora  Core  five  operating 
system. 


Figure  3.  Experimental  Set-up  for  Pilot  Study  1 


Procedures 

Participants  were  first  provided  an  overview  of  the  study  and  shown  the  Post-Trial  Questionnaire 
in  order  to  set  a  frame  of  reference  for  evaluating  the  display  concepts.  After  signing  the 
Informed  Consent  Document,  participants  filled-out  a  Background  Questionnaire  (Appendix  A) 
and  an  initial  Simulator  Sickness  Questionnaire  [16;  Appendix  B].  Although  sickness  symptoms 
were  not  anticipated  because  the  participants  were  experienced  with  video  games  that  had 
moving  backgrounds,  this  questionnaire  was  administered  before  and  after  each  trial  block  to 
document  health.  Training  consisted  of  demonstrating  the  four  fly-in  concepts  for  the  12  paired 
comparisons  for  the  first  drop  duration  time  trial  block.  Demonstrations  were  repeated  upon 
request.  Next,  experimental  trials  were  completed.  Note  that  there  was  no  interaction  required 
from  the  participant  except  to  reply  affirmatively  when  asked  if  he/she  was  ready  to  begin.  Each 
trial  consisted  of  showing  one  fly-in  concept  and  then  the  second.  (All  fly-ins  were  preceded  by 
a  fly-out,  as  described  above).  Immediately  after  viewing  each  pair  of  fly-in  concepts, 
participants  were  asked  to  compare  them  in  terms  of  situation  awareness,  visual  appeal,  and 
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preference  in  a  Post-Trial  Questionnaire  (Appendix  C).  A  sample  question  is  shown  in  Figure  4 
with  each  “fly-in”  referred  to  as  a  “transition.”  The  scoring  method  used  was  as  follows:  a  score 
of  “one”  was  given  if  the  concepts  were  equal.  If  the  first  fly-in  (transition)  viewed  was  better 
than  the  second,  a  positive  rating  was  given  with  two  being  “slightly  more,”  and  three  or  four 
being  “substantially  more.”  If  the  second  viewed  fly-in  was  better,  a  rating  of  negative  two, 
negative  three,  or  negative  four  was  given.  This  type  of  question  (using  judgment  matrices  to 
assess  relative  comparisons  of  pairs  of  experimental  conditions)  was  used  frequently  in  the 
studies  documented  in  this  report  and  is  based  on  the  Subjective  Workload  Dominance 
(SWORD)  technique  developed  by  [17].  The  ratings  used  in  the  statistical  analyses  were 
calculated  using  a  geometric  means  approach. 


3.  In  terms  of  preference,  compare  the  two  transitions. 


Mark  1  of  the  3  boxes  helow: 


Equal  preference  for 
Transition  I  and 
Transition  2 

□ 

If  yon  preferred  one  transition 
more  than  the  other,  how  mnch 
more  did  yon  prefer  it? 

Mark  I  of  the  3  colnmns  helow. 

Prefer  Transition  I 

□ 

then 

Slightly 

more 

> 

□ 

» 

□ 

Substantially 

more 

»> 

□ 

Prefer  Transition  2 

□ 

Comments  (If  yon  preferred  one  transition  over  the  other,  please  discnss  why): 


Fignre  4.  Sample  Qnestion  after  Participants  Viewed  a  Pair  of  Fly-ins  (transitions) 

After  each  drop  duration  time  block  of  12  trials,  a  Post-Block  Questionnaire  (Appendix  C)  was 
administered  asking  participants  to  make  a  series  of  relative  judgments,  comparing  each  fly-in 
concept  to  the  others  on  a  nine-point  scale.  Participants’  ratings  on  the  drop  duration  time  used 
in  that  particular  trial  block  were  also  collected  with  a  seven-point  Likert  Scale  (“Too  Slow”  . . . 
“Too  Fast”).  The  Post-Session  Questionnaire  was  administered  after  all  trials  were  completed 
(Appendix  C)  and  consisted  of  a  series  of  paired  comparisons  to  evaluate  drop  duration  time  and 
fly-in  concepts.  This  was  followed  by  various  questions  on  preferences  and  strategies  and  a 
section  for  general  comments. 


Results 

Analyses  were  conducted  on  the  subjective  data  collected  via  the  questionnaires.  These  results 
are  presented  below,  grouped  by  when  the  questionnaires  were  administered:  post-trial,  post¬ 
block  (after  all  trials  with  one  drop  duration  time),  and  post-session.  It  should  be  noted  that  the 
data  from  the  Simulator  Sickness  Questionnaire  was  also  analyzed  and  indicated  that  the 
participants  did  not  experience  any  significant  symptoms  as  a  result  of  viewing  any  of  the  fly-in 
concepts. 
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Post-Trial  Questionnaire 


Situation  Awareness  Rating,  Figure  5  presents  the  geometric  means  of  the  four  fly-in  concepts 
(Short  Vertical  Drop  (SVD),  Long  Vertical  Drop  (LVD),  Exponential  Sweep  (ES),  and  Linear 
Sweep  (LS))  for  each  block  of  drop  duration  time  (two,  four,  and  six  seconds).  Since  the 
experimental  trials  were  blocked  by  drop  duration  time,  Eriedman  Two-Way  Analyses  of 
Variance  were  first  performed  separately  on  paired  comparisons  data  from  each  time  block. 
Results  for  each  drop  duration  block  are  as  follows: 

2  second  block:  participant  ratings  did  not  significantly  differ  across  fly-in  concepts 
(/^(3)=  1.850,;?  =  0.604). 

4  second  block:  significant  difference  found  across  geometric  means  (x  (3)  =  1  L00,/>  = 
0.012).  Post-hoc  Wilcoxon  Signed  Ranks  Test  results  showed  that  ratings  for  the  SVD  concept 
were  lower  than  two  concepts  (Z=  2.201,;?  =  0.028;  Z=  1.992,;?  =  0.046  for  LS  and  ES, 
respectively).  Also,  LVD  was  rated  lower  than  LS  (Z=  2.201, =  0.028). 

6  second  block:  significant  difference  found  across  geometric  means  (/  (3)  =  1 1.300, = 
0.010).  Post-hoc  Wilcoxon  Signed  Ranks  Test  results  showed  that  comparison  ratings  for  SVD 
were  significantly  lower  than  LVD,  LS,  and  ES  (Z=  2.023,;?  =  0.043;  Z  =2.201, ;?  =  0.028;  Z  = 
2.201, =  0.028;  respectively). 


□  SVD 
■  LVD 
HLS 

□  ES 


2  sec.  4  sec.  6  sec. 


Figure  5.  Geometric  Mean  for  Situation  Awareness  Rating 
for  each  Fly-in  Concept  and  Drop  Duration  Time 
(error  bars  are  SEM:  standard  error  of  the  mean) 

Analysis  of  situation  awareness  paired  comparison  ratings  for  drop  duration  time  was  conducted 
using  the  Eriedman  Two-Way  Analysis  of  Variance  and  no  significant  differences  in  the 
geometric  means  were  found  (x  (2)  =  2.583, =  0.275).  The  ratings  were  then  collapsed  across 
drop  duration  times  and  reanalyzed  using  the  Eriedman  Two-Way  Analysis  of  Variance  and  a 
significant  difference  was  found  (x  (3)  =  1 1 .00,  p  =  0.012;  see  Eigure  6).  The  post-hoc 
Wilcoxon  Signed  Ranks  Test  results  showed  that  geometric  means  for  SVD  were  less  than  for 
LS  and  ES  (Z  =  2.201, =  0.028  for  both  comparisons). 
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Visually  Appealing  Rating,  Figure  7  presents  the  geometric  means  of  the  four  fly-in  concepts 
for  each  block  of  drop  duration  time.  Since  the  experimental  trials  were  blocked  by  drop 
duration  time,  Friedman  Two-Way  Analyses  of  Variance  were  first  performed  separately  on 
paired  comparisons  data  from  each  time  block.  Results  for  each  drop  duration  block  are  as 
follows: 

2  second  block:  participant  ratings  did  not  significantly  differ  across  fly-in  concepts 
(x'(3)  =  3.400,;?  =  0.334). 

4  second  block:  significant  difference  found  across  geometric  means  (x^(3)  =  8.300,;?  = 
0.040).  Post-hoc  Wilcoxon  Signed  Ranks  Test  results  showed  that  ratings  of  the  SVD  concept 
were  lower  than  the  LS  and  ES  concepts  (Z  =  2.023,  p  =  0.043  for  both  comparisons).  Also, 
LVD  was  rated  lower  than  LS  (Z  =  2.023, =  0.043). 

6  second  block:  participant  ratings  did  not  significantly  differ  across  fly-in  concepts 
(X'(3)  =  5.100,;?  =  0.165). 
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Figure  6.  Situation  Awareness  Geometric  Mean  for  each  Fly-in  Concept 
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Figure  7.  Geometric  Mean  for  Visually  Appealing  Rating 
for  each  Fly-in  Concept  and  Drop  Duration  Time 
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Similar  to  results  pertaining  to  the  situation  awareness  ratings,  no  significant  differences  were 
found  using  the  Friedman  Two-Way  Analysis  of  Variance  on  the  drop  duration  time  factor  (x^(2) 
=  4.33,  />  =  0. 1 15).  The  ratings  were  then  collapsed  across  drop  duration  times  and  reanalyzed 
using  the  Friedman  Two-Way  Analysis  of  Variance.  The  difference  across  geometric  means  just 
missed  being  statistically  significant  at  the  0.05  level  (x  (3)  =  7.450, =  0.059).  A  post-hoc 
Wilcoxon  Signed  Ranks  Test  was  performed  and  the  results  showed  that  SVD  geometric  means 
were  less  than  the  LS  and  ES  fly-in  concepts  (Z=  2.023, />  =  0.043;  Z=  2.201, =  0.028;  for  LS 
and  ES,  respectively).  Eigure  8  shows  the  geometric  means  for  the  visually  appealing  ratings  for 
each  fly-in  concept. 


Figure  8.  Visually  Appealing  Rating  Geometric  Mean  for  each  Fly-in  Concept 

Preference  Rating,  Eigure  9  presents  the  geometric  means  of  the  four  fly-in  concepts  for  each 
block  of  drop  duration  time.  Friedman  Two-Way  Analyses  of  Variances  were  first  performed 
separately  on  paired  comparisons  data  from  each  time  block.  Results  for  each  drop  duration 
block  are  as  follows: 

2  second  block:  participant  ratings  did  not  significantly  differ  across  fly-in  concepts 

(X'(3)  =  3.000,;?  =  0.392). 

4  second  block:  significant  difference  found  across  geometric  means  (x^(3)  =  13.400, = 
0.004).  Post-hoc  Wilcoxon  Signed  Ranks  Test  results  showed  that  ratings  of  the  SVD  concept 
were  lower  than  the  ES  and  ES  concepts  (Z  =  2.201,  =  0.028  for  both  comparisons).  Also, 

EVD  was  rated  lower  than  ES  (Z  =  2.201, =  0.028). 

6  second  block:  significant  difference  found  across  geometric  means  (x^(3)  =  9.400,;?  = 
0.024).  Post-hoc  Wilcoxon  Signed  Ranks  Test  Results  showed  that  ratings  of  the  SVD  concept 
were  lower  than  the  ES  and  ES  concepts  (Z  =  2.201, =  0.028;  Z  =  2.201, =  0.028  for  LS  and 
ES,  respectively). 

Similar  to  results  pertaining  to  the  situation  awareness  and  visual  appeal  ratings,  no  significant 
differences  were  found  using  the  Eriedman  Two-Way  Analysis  of  Variance  on  the  drop  duration 
time  factor  (x  (2)  =  3.5S3,p  =  0.167).  The  ratings  were  then  collapsed  across  drop  duration 
times  and  reanalyzed  using  the  Eriedman  Two-Way  Analysis  of  Variance  and  a  significant 
difference  was  found  (x  (3)  =  1 1.000, =  0.012).  A  post-hoc  Wilcoxon  Signed  Ranks  Test  was 
performed  and  the  results  showed  that  the  geometric  means  for  SVD  were  less  than  that  of  the 
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LS  and  ES  fly-in  concepts  (Z=  2201, p  =  0.028  for  both  comparisons).  Figure  10  shows  the 
geometric  means  of  the  preferenee  ratings  for  each  fly-in  concept. 


Figure  9.  Geometric  Mean  for  Preference  Rating  for  each  Fly-in  and  Drop  Duration  Time 


Figure  10.  Preference  Rating  Geometric  Mean  for  each  Fly-in  Concept 
Post-Block  Questionnaire 

After  all  trials  with  one  drop  duration  time  were  condueted,  participants  completed  a  post-block 
questionnaire  whieh  asked  whieh  of  the  fly-in  eoncepts  they  preferred,  as  well  as  their 
assessment  of  the  drop  duration  time.  The  following  reports  the  results  of  these  analyses. 

Preference  Rating.  Figure  1 1  presents  the  geometrie  means  of  the  four  fly-in  eoneepts  for  each 
block  of  drop  duration  time.  Friedman  Two-Way  Analyses  of  Varianees  were  flrst  performed 
separately  on  paired  comparisons  data  from  each  time  bloek.  Results  for  eaeh  drop  duration 
bloek  are  as  follows: 

2  second  bloek:  the  differenee  aeross  geometrie  means  just  missed  being  statistieally 
significant  at  the  .05  level  (x^(3)  =  7.620, =  0.055).  Post-hoe  Wileoxon  Signed  Ranks  Test 
results  did  not  show  any  statistieally  significant  differences. 
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2 

4  second  block:  significant  difference  found  across  geometric  means  (x  (3)  =  10.920,/?  = 
0.012).  Post-hoc  Wilcoxon  Signed  Ranks  Test  Results  showed  that  ratings  of  the  SVD  concept 
were  lower  than  that  for  the  LS  and  ES  concepts  (Z  =  2.023,  p  =  0.043;  Z  =  2.023,  /?  =  0.043). 

6  seeond  block:  significant  difference  found  across  geometric  means  (x^(3)  =  10.680,/?  = 
0.014).  Post-hoc  Wilcoxon  Signed  Ranks  Test  Results  showed  that  the  SVD  coneept  was 
preferred  less  than  the  LS  and  ES  concepts  (Z  =  2.023,/?  =  0.043  for  both  comparisons).  Also, 
ES  was  preferred  over  LVD  (Z=  2.201,/?  =  0.028). 
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Figure  11.  Geometric  Mean  for  Post-Block  Preference  Rating 
for  each  Fly-in  Concept  and  Drop  Duration  Time 

No  significant  differences  were  found  using  the  Eriedman  Two-Way  Analysis  of  Varianee  on  the 
drop  duration  time  factor  (x  (2)  =  1.583,/?  =  0.453).  The  ratings  were  then  collapsed  across  drop 
duration  times  and  reanalyzed  using  the  Eriedman  Two-Way  Analysis  of  Variance  and  a 
significant  difference  was  found  (x^(3)  =  9.000,/?  =  0.029).  A  post-hoe  Wilcoxon  Signed  Ranks 
Test  showed  that  the  geometric  means  for  the  SVD  concept  were  less  than  the  LS  and  ES  fly-in 
concepts  (Z  =  2.201, /?  =  0.028  andZ=  1.992,/?  =  0.046,  respectively).  Also,  ES  was  preferred 
over  LVD  (Z=  2.201, /?=0. 028).  Eigure  12  shows  the  geometric  means  for  the  post-block 
preference  ratings  by  fly-in  concept. 
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Figure  12.  Post-Block  Preference  Rating  Geometric  Mean  for  each  Fly-in  Concept 
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Drop  Duration  Time  Rating,  The  Post-Block  Questionnaire  included  a  seven-point  Likert 
Scale  for  participants  to  rate  their  judgment  of  the  duration  of  the  drop  time  for  the  fly-in  (Rating 
One  equals  “Too  Slow”  and  Rating  Seven  equals  “Too  Fast”).  A  Friedman  Two-Way  Analysis 
of  Variance  did  not  show  any  significant  differences  as  a  function  of  drop  duration  time  (x^(3)  = 
2333, p  =  0.31 1).  Data  from  this  question  are  shown  in  Figure  13.  The  horizontal  line  indicates 
the  value  if  the  participants  rated  the  duration  “just  right”  (i.e.,  the  middle  of  the  scale,  a  rating  of 
“four”).  A  comparison  of  the  mean  ratings  with  this  benchmark  suggests  that  participants  rated 
the  two  and  four  second  drop  duration  time  as  too  fast  and  the  six  second  drop  duration  time  as 
slightly  too  slow. 


5  2  sec.  4  sec.  6  sec. 


Figure  13.  Ratings  on  the  Drop  Duration  Time 


Post-Session  Questionnaire 

There  were  two  key  items  in  the  Post-Session  Questionnaire,  a  series  of  paired  comparisons 
asking  participants  to  compare  their  preferences  in  terms  of  fly-in  concept  and  drop  duration 
time.  Results  for  each  of  these  questions  will  be  addressed,  in  turn. 

Fly-in  Concept  Preference:  Results  from  a  Friedman  Two-Way  Analyses  of  Variance  test 
indicated  a  significant  difference  across  ratings  (x  (3)  =  1 1.640,/?  =  0.009).  The  preference 
ratings  were  lower  for  the  SVD  fly-in  compared  to  the  other  three  fly-in  concepts  (Figure  14). 
Post-hoc  Wilcoxon  Signed  Ranks  Tests  indicated  that  this  difference  was  significant  for  the  LS 
and  ES  fly-in  concepts  (Z=  2.023,  p  =  0.043  for  both  comparisons). 
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Figure  14.  Post-Session  Preference  Rating  Geometric  Mean  for  each  Fly-in  Concept 

Fly-in  Drop  Duration  Time  Preference,  A  Friedman  Two-Way  Analysis  of  Variance  was 
performed  on  paired  comparisons  data  from  relative  judgments  of  drop  time  preference,  and  the 
results  showed  that  the  differences  across  geometric  means  just  missed  being  statistically 
significant  at  the  0.05  level  (x^(2)  =  4.750,/?  =  0.093).  Results  from  a  post-hoc  Wilcoxon  Signed 
Ranks  Test  indicated  that  preference  ratings  for  four  seconds  were  more  favorable  than  ratings 
for  two  seconds  (Z=  2.201,/?  =  0.028;  Figure  15). 


Figure  15.  Post-Session  Preference  Rating  Geometric  Mean  for  each  Drop  Duration  Time 

Conclusions 

The  results  were  consistent  across  questionnaires  administered  after  each  individual  trial,  block 
of  trials,  and  at  the  end  of  the  experimental  session.  First,  there  were  no  significant  differences 
in  rating  between  the  two  vertical  drop  conditions,  indicating  that  the  extra  time  devoted  to  the 
second  phase  of  the  long  vertical  drop  (LVD)  was  not  beneficial.  More  importantly,  participants 
preferred  the  exponential  sweep  (ES)  and  the  linear  sweep  (LS)  over  both  the  short  and  long 
vertical  drop  concepts  (SVD,  LVD).  In  both  vertical  drop  fly-in  concepts,  the  pitch  of  the  virtual 
camera  did  not  change  during  the  drop  phase.  In  contrast,  the  pitch  increased  for  the  other  two 
fly-in  concepts,  one  linearly  (LS)  and  one  exponentially  (ES).  Increasing  the  pitch  during  the 
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drop  phase  lengthened  the  time  in  whieh  the  target  remained  in  the  eamera  FOV.  Review  of 
partieipants’  eomments  indieated  that  this  was  the  basis  of  their  preferenee  and  situation 
awareness  ratings.  Partieipants  did  not  want  to  lose  sight  of  the  target  during  the  fly-in. 

The  preferenee  ratings  suggest  that  the  drop  duration  time  should  be  four  seeonds.  This  duration, 
together  with  one  second  for  the  zoom  in  phase,  would  mean  allowing  five  seconds  for  the  total 
fly-in  segment  of  the  transition. 


Implications  for  Future  Research 

Based  on  the  results  of  Pilot  Study  1,  we  decided  that  the  fly-in  segments  to  be  tested  in  follow- 
on  studies  should  be  approximately  five  to  six  seconds  in  duration.  Furthermore,  the  virtual 
camera’s  view  should  increase  in  pitch  during  the  fly-in  (similar  to  what  was  employed  in  the 
linear  and  exponential  sweep  conditions).  It  is  also  recommended  that  the  camera  needs  to  spend 
as  much  time  as  possible  at  a  location  that  provides  maximum  view  of  the  area.  These  design 
criteria  are  supported  by  the  participants’  comments  to  keep  the  target  in  view  as  much  as 
possible,  but  not  lengthen  the  total  fly-in  duration  any  more.  Some  participants  suggested 
providing  more  time  (even  a  pause)  at  the  top  of  the  fly-in. 

For  follow-on  pilot  studies  of  the  fly-in,  we  also  decided  not  to  present  the  fly-out  segment  as 
well.  In  the  present  study,  fly-out  transition  segments  preceded  each  fly-in.  It  was  originally 
hypothesized  that  the  presence  of  a  fly-out  might  help  the  participants  divorce  themselves  from 
the  environment  in  the  previous  camera  view.  However,  pilot  study  participants  commented  that 
the  presence  of  the  fly-out  was  distracting  in  the  evaluation  process. 

This  Pilot  Study,  besides  providing  data  to  drive  fly-in  design  for  transitioning  between  camera 
views,  was  informative  in  regards  to  evaluation  methodology.  First,  the  hardware  utilized  in  this 
study  imposed  several  limitations.  The  database  was  slow  in  rendering  the  transition  segments. 
This  meant  there  was  a  short  break  between  the  viewing  of  each  fly-in  concept  for  the  paired 
presentations,  making  it  more  difficult  for  participants  to  compare  the  pairs  of  fly-ins  back  to 
back.  Although  the  experimental  variables  were  presented  in  all  possible  orders  across  trials,  to 
control  for  order  effect,  the  breaks  imposed  by  the  hardware  may  have  introduced  fatigue  and/or 
frustration  in  the  completion  of  the  experimental  sessions.  It  is  desirable  for  follow-on  research 
to  employ  different  hardware,  such  that  the  delay  between  viewing  transition  segments  is 
minimized. 

This  study  also  showed  the  shortcomings  of  the  data  we  measured.  With  the  approach  used  in 
Pilot  Study  1,  we  were  limited  to  subjective  ratings  of  the  various  transition  parameter 
manipulations.  It  is  more  desirable  to  have  an  approach  that  is  a  better  representation  of  the 
envisioned  operational  task  after  changing  camera  views.  Moreover,  objective  measures  of 
situation  awareness  and  the  degree  to  which  negative  transfer  of  context  occurs  would  be 
desirable. 
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PILOT  STUDY  2 


Purpose 

In  Pilot  Study  1  we  examined  several  different  fly-in  eoneepts  and  durations  to  determine  what 
parameters  were  best  for  implementing  this  segment  of  a  transition  format  designed  to  aid 
ehanging  eamera  views.  With  these  parameters,  we  designed  three  improved  fly-in  eoneepts 
using  new  software  tools.  One  objective  of  Pilot  Study  2  was  to  evaluate  these  three  fly-in 
concepts.  Another  objective  was  to  obtain  an  objective  measure  of  situation  awareness  afforded 
by  the  transition  segment,  in  addition  to  subjective  assessments.  In  particular,  we  were  interested 
in  whether  situation  awareness  of  points  of  interest  beyond  the  sensor  view  would  be  enhanced 
with  these  fly-in  concepts. 


Fly-in  Development 

Several  major  changes  were  made  during  the  development  of  the  second  generation  of  fly-ins. 
The  first  was  a  change  in  the  focus  of  the  fly-ins.  The  first  iteration  of  fly-in  segments  focused 
on  the  area  surrounding  the  UAV  with  the  idea  that  this  would  give  the  operator  a  better  idea  of 
the  environment  as  it  relates  to  the  UAV,  and  that  it  would  allow  the  operator  to  see  the  sensor’s 
entire  area  of  influence.  After  reviewing  the  fly-ins  from  the  first  study  and  consultation  with 
UAV  operators,  we  determined  that  this  was  not  the  best  use  of  the  GITZ  transition  for  several 
reasons.  First,  the  purpose  of  the  fly-in  was  to  provide  operators  with  the  visual  cues  needed  to 
rapidly  develop  a  cognitive  map  of  spatial  points  of  interest  relative  to  the  environment  in  the 
sensor  view,  not  the  environment  surrounding  the  UAV.  Information  about  the  environment 
surrounding  the  UAV  is  already  represented  in  the  Tactical  Situation  Display.  Also,  the  UAVs 
that  are  being  simulated  do  not  fly  directly  above  their  target,  as  this  would  give  away  their 
position  and  intentions.  Instead  they  fly  2-4  NM  away  from  the  area-of- interest  (AO I),  at  a  range 
that  is  best  for  the  sensors  and  that  puts  the  AOI  off  to  the  side  of  the  UAV’s  loiter.  With  this  in 
mind,  we  changed  the  center  point  at  the  beginning  of  the  fly-in  from  the  area  around  the  UAV 
to  the  area  around  the  target.  By  incorporating  this  with  a  new  stare  point  lock-on  tool,  we  were 
able  to  keep  the  target  in  the  center  of  the  camera  view  at  all  times  regardless  of  the  fly-in  path. 
This  approach  allowed  for  much  more  flexibility  in  the  path  design. 

Changing  the  focus  of  the  fly-in  from  the  area  surrounding  the  UAV  to  the  area  surrounding  the 
target  required  us  to  reexamine  the  Virtual  Camera’s  (VC)  start-point  at  the  beginning  of  the  fly- 
in.  The  three  main  ideas  were:  1)  start  the  VC  over  the  target,  2)  keep  the  VC  over  the  UAV,  or 
3)  start  the  VC  above  the  UAV  away  from  the  target.  Figure  16  illustrates  these  three 
alternatives.  Dropping  the  camera  straight  down  into  the  UAV  (Number  2  in  Figure  16)  did  not 
make  sense  as  it  was  designed  with  the  idea  that  the  GITZ  transition  would  focus  on  the  area 
surrounding  the  UAV.  Starting  the  VC  above  the  target  (Number  1  in  Figure  16)  would  tailor 
the  fly-in  to  the  new  focal  point,  the  target.  This  approach  would  give  the  operator  a  bird’s-eye- 
view  of  the  target  area  before  flying  into  the  sensor  view.  However,  an  expert  in  the  role  of 
perspective  in  observation  and  navigation  pointed  out  that  this  was  an  unnatural  flight  path  for  a 
human  to  see.  The  combination  of  the  straight  downward  fall  and  drastic  change  in  perspective 
would  give  the  sensation  of  falling  backwards  and  make  processing  information  in  the  camera 
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view  very  difficult.  He  proposed  the  concept  illustrated  in  Number  3  of  Figure  16:  starting  the 
VC  behind  and  above  the  actual  sensor.  This  would  allow  the  VC  to  fly  towards  the  target 
throughout  the  fly-in  instead  of  falling  on  it.  This  fly-in  would  be  visually  similar  to  flying  a 
plane  towards  the  target. 


Figure  16.  Illustration  of  Three  Possible  Virtual  Camera  Start  Points 

Using  this  approach  (starting  the  VC  above  and  behind  the  target),  we  developed  three  new  fly-in 
concepts:  linear,  shallow  curve,  and  deep  curve  (Figure  17).  For  the  linear  fly-in,  we  simply  took 
the  vector  from  the  target  through  the  UAV  and  had  the  VC  start  12,000  ft  away  from  the  UAV 
on  the  opposite  side  of  the  vector  from  the  target.  We  had  the  UAVs  start  at  an  altitude  of 
4500  ft  AGL  (standoff  distances  from  the  ground  targets  were  adjusted  to  maintain  camera  pitch 
of  ~45  degrees).  This  meant  that  the  VC  started  at  an  altitude  of  only  13,500  ft  AGL.  We  were 
able  to  get  the  entire  area  we  wanted  to  observe  in  the  VC’s  view  at  the  start  point  by  increasing 
the  FOV  to  72  degrees.  This  fly-in  was  broken  up  into  two  stages:  the  VC’s  descent  from  the 
start  point  to  the  UAV,  and  the  zoom-in  to  the  sensor’s  viewpoint.  The  first  portion  lasted  5.15 
seconds  and  used  an  exponential  function  of  0.6  so  the  fly  in  started  slowly  and  sped  up  in  the 
middle.  The  second  portion  of  the  fly-in  lasted  0.85  seconds  and  used  an  exponential  function  of 
negative  0.9  to  slow  down  the  zoom-in  as  it  approached  the  end. 

The  two  curved  fly-ins  (see  Figure  17)  started  12,000  ft  away  from  the  UAV  as  well  and  had  72 
degree  FOVs.  These  two  fly-ins  always  started  at  an  angle  of  77.5  degrees  above  horizontal  with 
respect  to  the  UAV,  whereas  the  first  fly-in  started  at  the  same  angle  that  the  UAV  was  at  with 
respect  to  the  target  (in  this  study  it  was  ~46  degrees  in  all  trials).  The  curved  fly-ins  flew 
towards  the  target  along  paths  created  using  cubic  Bezier  curves.  This  was  done  with  the 
thought  that  providing  the  operator  with  multiple  perspectives  of  the  area  would  give  them 
information  about  its  layout.  One  fly-in  followed  a  shallow  curve  that  approached  the  vector 
from  the  target  through  the  UAV  as  it  flew.  The  first  stage  of  this  fly-in  took  5.2  seconds  with  an 
exponential  function  of  0.2.  The  second  stage  of  the  shallow  curve  took  0.8  seconds  and  used  an 
exponential  function  of  negative  2.0.  The  other  fly-in  followed  a  deeper  curve  that  went  well 
beyond  the  vector  and  approached  horizontal  flight  as  it  flew  in  towards  the  camera.  The  first 
stage  of  this  fly-in  took  4.8  seconds  with  an  exponential  function  of  0. 1 .  The  second  stage  of  the 
deep  curve  took  1.2  seconds  and  used  an  exponential  function  of  negative  3.0. 
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The  exponential  functions  that  we  used  were  chosen  for  several  reasons.  The  first  was  to  give 
more  time  at  the  beginning  of  the  fly-in  to  view  the  whole  area.  The  second  was  to  slow  down 
the  camera  before  the  end  of  the  fly-in  so  the  conclusion  would  not  be  abrupt.  The  third  reason 
was  to  give  the  impression  that  the  cameras  moved  at  similar  speeds  during  the  different  fly-in 
concepts.  In  examining  the  different  combinations  of  stage  durations  and  exponential  functions, 
we  found  that  the  parameters  specified  above  were  the  best.  All  three  fly-in  concepts  had  a  total 
duration  of  six  seconds,  meeting  the  recommendation  derived  from  Pilot  Study  1 . 


. 

Linear  Fly-in 

Shallow  Curve  Fly-in 

Deep  Curve  Fly-in 

Figure  17.  Illustrations  of  Three  Fly-in  Concept  Camera  Paths 


Method 


Experimental  Design 

Twelve  volunteers  (mean  age  =  25.75)  participated  in  this  two-part  study.  Part  I  of  Pilot  Study  2 
was  designed  to  collect  an  objective  measure  of  the  effect  of  the  fly-in  concept  on  situation 
awareness  of  the  AOI,  in  addition  to  a  subjective  assessment.  Part  2  used  procedures  similar  to 
those  used  in  Pilot  Study  1  and  was  designed  to  collect  additional  subjective  assessments  of  the 
improved  fly-ins. 

In  Part  I,  we  used  a  3  x  2  within-subjects  design;  two  trials  with  each  of  the  three  fly-in 
concepts:  linear,  shallow  curve,  and  deep  curve.  Trials  were  blocked  by  the  three  fly-in 
concepts  and  the  trial  orders  in  which  the  fly-in  concepts  were  presented  to  the  participants  was 
counterbalanced  across  participants.  The  UAV’s  2.55  NM  x  2.55  NM  target  area  was  divided  up 
into  nine  sections.  The  area,  across  the  sections,  was  populated  with  overlaid  computer¬ 
generated  symbology  consisting  of  4-5  red  (enemy)  flags,  4-5  blue  (friendly)  flags,  and  one  flag 
of  each  of  the  following  colors;  light  green,  light  blue,  magenta,  yellow,  and  orange.  The  blue 
and  red  flags  acted  as  clutter  with  the  rest  being  possible  targets.  The  flags  were  distributed  such 
that  each  column  and  row  of  sections  in  the  area  contained  3-5  flags  with  at  least  one  flag  in  each 
section  and  no  more  than  two  flags  of  the  same  color  in  any  section.  The  UAV  was  positioned 
approximately  5000  feet  above  and  away  from  the  center  of  the  area  with  its  sensor  focused  on  a 
target  at  the  center  point.  See  Figure  18  for  an  illustration  of  the  placement  of  the  colored  flags. 
Note;  the  yellow  grid  lines  were  not  visible  during  trials. 
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In  Part  II  of  the  study,  conducted  after  completion  of  Part  I,  the  fly-ins  were  tested  using  a 
balanced  paired  comparison  design.  To  control  for  order  bias,  each  fly-in  concept  was  compared 
to  every  other  fly-in  concept,  with  each  concept  being  presented  first  in  one  comparison  and 
presented  second  in  the  second  comparison.  After  each  pair  of  fly-ins  were  shown  back-to-back, 
participants  were  asked  to  complete  a  questionnaire  to  collect  subjective  assessment  on  the 
degree  to  which  one  was  better  than  the  other  in  terms  of  situation  awareness  and  visual  appeal. 

In  contrast  to  the  procedure  used  in  Pilot  Study  1,  the  presentations  of  the  fly-in  transition 
segments  were  not  preceded  with  a  fly-out  transition  segment.  The  order  of  the  comparisons  was 
randomly  determined  with  the  constraint  that  there  were  no  more  than  two  consecutive  trials  with 
the  same  first  fly-in  concept. 


Figure  18.  Illustrations  of  Method  used  to  Distribute  the  Presentation  of  Colored  Flags 


Simulation  Environment 

The  study  was  performed  using  AFRL’s  Open  Scene  Environment  (OSE)  version  0.4.67 
visualization  software  to  present  the  participant  with  a  manipulated  synthetic  camera  view  that 
moved  along  a  preset  path  in  an  urban  environment.  The  computer  used  for  this  experiment  was 
a  Dell  Precision  Workstation  670,  Pentium  Xeon  with  dual  3.6GHz  processors,  1GB  RAM,  and  a 
512MB  PCIe  nVidia  Quadro  EX  video  card.  A  single  17”  flat  screen  LCD  monitor  was  used. 

The  participant  viewed  the  monitor  while  the  experimenter  controlled  experimental  conditions 
with  a  keyboard  and  mouse.  Ely-in  concepts  were  shown  using  the  Open  Scene  Environment 
and  command-line  arguments  in  the  Eedora  Core  5  operating  system.  Back-to-back  comparison 
videos  were  shown  using  Windows  Media  Player,  which  helped  reduce  the  delay  between 
viewing  pairs  of  transition  segments. 
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Procedures 


Participants  were  first  provided  an  overview  of  the  study  and  shown  the  Post-Trial  Questionnaire 
(Appendix  D)  in  order  to  set  a  frame  of  referenee  for  evaluating  the  display  eoneepts.  After 
signing  the  Informed  Consent  Doeument,  partieipants  filled  out  Baekground  Questionnaire 
(Appendix  A)  and  an  initial  Simulator  Siekness  Questionnaire  [16;  Appendix  B],  Even  though 
sickness  symptoms  were  not  anticipated  beeause  the  partieipants  were  experieneed  with  video 
games  that  had  moving  baekgrounds,  this  questionnaire  was  administered  before  and  after  eaeh 
trial  bloek  to  doeument  health.  Training  eonsisted  of  demonstrating  eaeh  of  the  fly-ins 
individually  and  explaining  the  charaeteristics  of  the  fly-in  paths.  Praetice  trials  were  eonducted 
until  participants  felt  eomfortable  with  the  proeedures.  Then  experimental  trials  were  eompleted. 

In  Part  I  of  the  study,  partieipants  were  instrueted  to  ereate  a  mental  map  of  the  area  and  its  flags 
during  the  fly-in.  Following  the  presentation  of  the  fly-in,  they  were  asked  to  draw  a  line  from 
the  eenter  of  the  area  to  the  loeation  of  a  flag  that  was  then  specified  (light  green,  light  blue, 
magenta,  yellow,  or  orange)  on  a  form  (Figure  19;  Appendix  D).  The  participants  were  not  told 
whieh  flag  to  indieate  until  after  the  fly-in  was  presented.  The  angle  formed  by  the  interseetion 
of  the  recalled  vector  of  the  target  and  the  actual  vector  from  the  eenter  of  the  area  was  used  to 
measure  aeeuraey  from  0  to  180  degrees. 


1 )  For  this  trial,  draw  a  line  from  the  center  flag  to  the  direction  of  the  (light  green,  light 
blue,  magenta,  yellow,  and  orange)  flag: _ 


c. 

V 

Figure  19.  Post-Trial  Form  for  Participants  to  Indicate  the  Direction  of  the  Requested  Flag 

Next,  partieipants  rated  the  fly-in  for  situation  awareness  and  visual  appeal  using  Fikert  Seales 
(Figure  20;  Appendix  D).  The  first  trial  with  each  type  of  fly-in  was  treated  as  a  training  trial. 
Only  data  from  the  seeond  trial  with  eaeh  fly-in  were  analyzed. 
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Mark  the  box  (□)  which  best  reflects  your  ratings  for  each  of  the  following: 

2)  Situation  Awareness:  For  this  trial,  rate  your  overall  situation  awareness  —  the  degree 
to  whieh  you:  were  aware  of  important  elements  in  your  environment  (i.e.,  landmarks,  targets, 
major  features),  eomprehended  your  eurrent  loeation,  and  were  able  to  projeet  future  status. 


Low 

Situation  Awareness  Rating 

High 

Never 

Always 

Aware 

Aware 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

3)  Visually  Appealing:  For  this  trial,  rate  how  visually  appealing  the  fly-in  was. 


Visually  Appealing  Rating 

Lowest 

Appeal 

Highest 

Appeal 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Figure  20.  Post-Trial  Question  -  Part  1  for  Rating  Fly-in  Situation  Awareness  and  Visual  Appeal 


After  the  six  trials  of  Part  I  were  eompleted  (two  with  eaeh  fly-in),  partieipants  were  asked  to 
fill-out  a  Post-Part  1  Questionnaire  (Appendix  D)  to  indieate  whieh  fly-in  approaeh  they 
preferred.  The  survey  eonsisted  of  a  series  of  relative  judgments.  Eaeh  judgment  required  the 
partieipant  to  indieate  whieh  of  the  two  eoneepts  being  eompared  was  preferred  in  terms  of 
situation  awareness  gained  or  visual  appeal.  A  nine-point  seale  was  used  for  eomparing  the 
preferenee  of  eaeh  of  the  pairs  of  fly-in  eoneepts. 

After  eaeh  trial  in  Part  II  of  Pilot  Study  2,  partieipants  were  asked  to  eompare  the  two  fly-in 
eoneepts  just  presented  baek-to-baek  to  indieate  whieh  one  provided  more  situation  awareness 
(Figure  21)  and  was  more  visually  appealing  (Figure  22)  on  a  Post-Trial-Part  2  Questionnaire. 
After  eompleting  all  six  trials  in  Part  II,  partieipants  were  asked  to  eomplete  a  Post-Part  2 
Questionnaire,  indieating  whieh  fly-in  eoneept  they  preferred.  These  questionnaires  ean  be 
viewed  in  Appendix  D.  All  subjeetive  data  were  analyzed  using  the  proeedures  employed  in 
Pilot  Study  1 . 

After  both  Part  I  and  II  were  eompleted,  study  partieipants  were  given  a  Post-Session 
Questionnaire  (Appendix  B)  in  whieh  they  were  asked  about  their  strategies  in  making  the 
eomparison  judgments.  The  questionnaire  also  solieited  other  ideas  for  fly-in  eoneepts  and 
general  eomments. 
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1.  In  terms  of  your  ability  to  sain  situation  awareness  (SA)  of  the  entire  area,  compare  the  first  and  second  fly-in  observed. 
SA  =  the  degree  to  which  you:  were  aware  of  important  elements  in  your  environment  (i.e.,  landmarks,  targets, 
major  features),  comprehended  your  current  location,  and  were  able  to  project  future  status. 


Mark  f  of  the  3  boxes  below: 


SA  eaual  with  Ffv-in  I  and  2 

□ 

If  you  had  more  SA  with  one  fly-m 
compared  to  the  other,  how  much 
more? 

Mark  I  of  the  3  columns  below. 

More  SA  with  Fly-in  1 

□ 

Slightly  more 

> 

□ 

Substantially 

more 

»> 

More  SA  with  Fly-in  2 

□ 

then: 

» 

□ 

□ 

Figure  21.  Post-Trial  Question  Comparing  Situation  Awareness  across  Fly-in  Concepts 


2.  In  terms  of  being  visually  avvealine,  compare  the  first  and  second  fly-in  observed. 


Mark  1  of  the  3  boxes  below: 


Fly-in  1  and  2  were  eaually  visually  appealing 

□ 

If  you  found  one  fly-m  more  visually 
appealing,  how  much  more  appealing  was 
it  compared  to  the  other  fly-in? 

Mark  1  of  the  3  columns  below. 

Fly-in  1  more  visually  appealing 

□ 

Slightly  more 

> 

□ 

Substantially 

more 

»> 

Fly-in  2  more  visually  appealing 

□ 

then: 

» 

□ 

□ 

Figure  22.  Post-Trial  Question  Comparing  Visual  Appeal  across  Fly-in  Concepts 


Results  -  Part  I 

The  average  differenee  in  the  angle  between  the  marked  location  of  the  requested  flag  and  the 
real  location  of  the  flag  (for  the  second  trial  with  each  fly-in  concept)  was  analyzed  in  a  one-way 
Analysis  of  Variance  (ANOVA).  Results  indicated  that  the  mean  accuracy  in  marking  the 
requested  flag  location  on  the  map,  across  fly-in  concepts,  just  missed  being  statistically 
significantly  different  at  the  0.05  level  (F(2,22)  =  3.485,^3  =  0.055).  Post  hoc  Hypothesis  Tests 
on  the  ANOVA  were  completed  to  compare  mean  accuracy  for  each  possible  pair  of  fly-in 
concepts  at  a  time.  The  mean  accuracy  for  the  shallow  curve  and  deep  curve  were  contrasted 
against  that  for  the  linear  fly-in.  Hypothesis  Test  results  showed  that  participants  more 
accurately  indicated  the  location  of  the  requested  flag  with  the  linear  fly-in  concept  as  it  was 
significantly  less  compared  to  performance  with  the  Shallow  and  Deep  Curve  (F(l,l  1)  =  12.634, 
j!7  =  0.005).  See  Figure  23. 
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Figure  23.  Accuracy  iu  ludicatiug  Target  Flag  Locatiou  with  Each  Fly-iu  Coucept 

Analyses  were  also  eondueted  on  the  subjective  questionnaire  data  collected  after  the  second  trial 
with  each  fly-in  concept  (Figure  20).  Friedman  Two-Way  Analysis  of  Variance  tests  were 
conducted  on  participants’  two  post-trial  rating  scales.  The  results  failed  to  find  any  significant 
differences  in  the  ratings  pertaining  to  situation  awareness  (y  (2)  =  1.167,/?  =  0.558;  Figure  24) 
and  visual  appeal  (x^(2)  =  2.042,/?  =  0.360;  Figure  25).  Using  the  same  statistical  test,  the  results 
also  did  not  show  a  significant  difference  in  participants’  rating  of  their  fly-in  concept  preference 
in  the  questionnaire  administered  after  all  trials  were  completed  in  Part  I  (x^(2)  =  0.042,/?  = 
0.979;  Figure  26).  Finally,  participants’  ratings  on  the  Simulator  Sickness  Questionnaire 
indicated  that  there  were  not  any  significant  symptoms  as  a  result  of  viewing  the  fly-in  concepts. 


Figure  24.  Post-Trial  Situatiou  Awareuess  Ratiug  for  Each  Fly-iu  Coucept 
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Figure  25.  Post-Trial  Visual  Appeal  Rating  for  Each  Fly-in  Concept 
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Figure  26.  Post-Study  Part  I  Geometric  Mean  Rating  for  Fly-in  Preference 


Results  -  Part  II 

Only  subjective  data  were  collected  in  Part  II  that  involved  the  paired  comparison  design. 
Analyses  using  the  Friedman  Two-Way  Analysis  of  Variance  failed  to  find  statistical  significant 
differences  in  participants’  post-trial  ratings  on  the  three  fly-in  concepts  in  terms  of  situation 
awareness  (x^(2)  =  0.875,/?  =  0.646)  and  visual  appeal  (x^(2)  =  2.042,  p  =  0.360).  Nor  were 
there  significant  differences  found  in  the  preference  ratings  collected  after  all  trials  were 
completed  in  Part  II  (x^(2)  =  0.042,  p  =  0.979)  and  in  a  Final  Questionnaire  (x^(2)  =  0.31 5,  p  = 
0.829). 


Conclusions 

Many  participants  commented  that  they  felt  the  linear  fly-in  provided  better  situation  awareness 
of  the  area,  but  that  both  curved  fly-ins  were  more  visually  appealing.  However,  analysis  of  the 
questionnaire  data  did  not  show  a  significant  difference  across  fly-in  concepts  in  terms  of 
perceived  situation  awareness,  visual  appeal,  or  fly-in  preference.  In  light  of  these  contradictory 
results,  we  will  base  the  conclusions  for  Pilot  Study  2  on  the  objective  performance  data. 
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For  the  task  in  which  participants  had  to  recall  where  the  requested  flag  was  located  and  indicate 
it  on  a  form  (Figure  19),  performance  was  best  when  the  linear  fly-in  was  utilized,  in  comparison 
to  the  two  curved  fly-ins.  Performance  with  the  linear  fly-in  was  both  more  accurate  (lower 
angle  difference)  and  consistent  (less  variance)  than  those  for  the  shallow  curve  and  deep  curve 
fly-in  concepts.  These  data  indicate  that  participants  were  able  to  more  consistently  and 
accurately  create  and  recall  mental  maps  of  the  areas  when  using  the  linear  fly-in.  We  surmise 
that  participants  were  able  to  create  better  cognitive  renditions  of  the  areas  with  the  linear  fly-in 
due  to  the  fixed  perspective  it  utilized.  With  the  linear  fly-in,  as  the  camera  flew  in,  the  flags’ 
orientations  relative  to  the  fly-in  path  vector  remained  fixed,  whereas  with  the  curved  fly-ins, 
their  orientations  changed  constantly.  It  should  be  noted  that  in  this  experiment  we  only 
measured  directional  accuracy  across  a  plane.  It  is  not  known  which  fly-in  concept  provided 
better  cues  with  respect  to  elevation  and  distance  differences. 


Implications  for  Future  Research 

Based  on  the  results  of  Pilot  Study  2,  we  decided  that  a  linear  fly-in,  as  opposed  to  a  curved 
approach,  should  be  tested  in  follow-on  studies.  Furthermore,  consideration  should  be  given  to 
the  numerous  participant  comments  in  determining  other  parameters  of  the  fly-in.  Many 
mentioned  that  they  would  like  the  virtual  camera  to  spend  more  time  at  the  beginning  or  top  of 
the  fly-in.  This  would  provide  more  time  to  survey  the  entire  area  and  create  a  mental  map.  In 
regards  to  the  middle  segment,  several  participants  said  that  it  should  be  speeded  up,  because 
they  did  not  pick-up  any  meaningful  information  during  this  portion.  At  the  end  of  the  fly-in, 
participants  indicated  that  there  should  be  a  less  abrupt  stop  by  slowing  the  transition.  The 
information  presented  at  the  end  of  the  fly-in  is  of  great  importance  as  well,  as  it  provides  the 
operator  with  information  about  the  target’s  immediate  surroundings.  In  sum,  these  comments 
suggest  that  the  very  beginning  and  end  of  the  fly-in  are  the  two  most  important  segments  for 
acquiring  information. 

Most  participants  also  expressed  disappointment  or  frustration  at  the  fact  that  they  were  unable  to 
control  the  fly-in.  It  was  suggested  to  have  a  function  which  allowed  the  operator  to  choose 
whether  or  not  they  wanted  to  use  the  GITZ  transition,  as  well  as  tools  for  controlling  parameters 
of  the  fly-in  (e.g.,  speed).  They  also  wanted  to  be  able  to  go  back  and  forth  and  pause  during  the 
fly-in  (similar  to  the  functions  on  many  digital  video  recorders). 
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FULL  MULTI-UAV  SIMULATION  EVALUATION 


Purpose 

Pilot  Studies  1  and  2  focused  on  the  fly-in  segment  of  a  transition  format.  The  participants’  only 
mission-related  task  was  to  view  the  GITZ  transition  and  the  data  collected  primarily  consisted 
of  subjective  assessments.  Only  one  objective  measure  was  recorded  (Pilot  Study  2),  the  average 
difference  in  the  angle  between  the  marked  location  of  the  requested  flag  and  the  real  location  of 
the  flag.  However,  these  part-task  simulations  were  valuable  in  that  a  series  of  fly-in  segments 
could  be  presented  rapidly,  allowing  multiple  fly-in  parameters  to  be  systematically  manipulated 
and  evaluated.  This  provided  data  to  inform  the  design  of  the  fly-in  segment  for  testing  in  a  full 
simulation  evaluation. 

One  purpose  of  the  present  evaluation  was  to  determine  if  participants’  subjective  assessments  of 
a  fly-in  approach  determined  from  the  results  of  the  pilot  studies  are  also  favorable  when  the 
transition  is  employed  in  a  full  multi-UAV  simulation  that  involves  the  completion  of  multiple 
tasks  while  periodically  switching  missions/camera  views.  Besides  recording  the  participants’ 
impressions,  another  objective  was  to  obtain  multiple  performance  measures  to  determine  if  the 
GITZ  transition  helped  the  participants’  overall  situation  awareness  and  improved  their 
performance  on  a  target  designation  task  after  switching  to  a  new  UAV/camera  view.  This 
evaluation  also  manipulated  the  mission  scenario  to  determine  if  the  utility  of  the  GITZ  transition 
depends  on  whether  the  previous  mission  was  a  static,  surveillance  type  mission  or  a  dynamic, 
close  air  support  mission.  Finally,  this  study  was  designed  to  determine  if  the  presence  of  the 
GITZ  transition  had  any  negative  effects  on  participants’  completion  of  secondary  mission- 
related  tasks.  This  is  important  because  if  the  GITZ  transition  degrades  performance  on  any 
task,  then  its  candidacy  for  multi-UAV  control  applications  is  questionable,  even  if  it  improves 
situation  awareness  after  switching  to  a  different  UAV/camera  view. 

Transition  Development 

The  full  multi-UAV  control  simulation  environment  used  in  this  evaluation  allowed 
scenarios  that  included  multiple  UAVs  flying  in  loiterers  patterns.  Because  the  UAVs  did  not 
have  to  be  stationary,  adjustments  were  required  to  make  the  fly-in  more  flexible.  The  camera 
pitch  was  no  longer  fixed  at  46.2  degrees  but  was  determined  by  the  angle  from  horizontal 
between  the  UAV  and  the  stare  point  on  the  ground.  Each  UAV’s  altitude  and  distance  from  the 
targets  changed  from  Pilot  Study  2  as  well  because  we  were  no  longer  constrained  by  the  virtual 
environment.  The  UAVs  flew  between  8,000  and  12,000  ft  MSL  at  2  to  4  NM  from  their  targets. 
This  allowed  us  to  return  to  the  more  realistic  48  degree  max  FOV  that  we  used  in  Pilot  Study  1. 
The  minimum  FOV  was  reduced  to  0.41  degrees  (spotter  mode)  to  allow  the  participants  to 
identify  and  designate  targets  on  the  ground.  Other  changes  were  made  in  the  linear  fly-in,  based 
on  comments  from  Pilot  Study  2.  For  instance,  we  took  the  vector  from  the  target  through  the 
UAV  and  had  the  VC  start  18,000  ft  away  from  the  UAV  on  the  opposite  side  of  the  vector  from 
the  target.  This  distance  was  increased  from  the  original  to  allow  the  entire  area  of  observation 
to  fit  in  the  reduced  max  FOV.  Also,  a  fly-out  segment  preceded  the  fly-in  for  this  study  to 
determine  if  it  would  help  reduce  negative  transfer  of  context  that  could  occur  when  switching 
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from  one  camera  view  to  another.  Figure  27  illustrates  how  the  GITZ  transition  was 
implemented  for  this  evaluation.  In  this  figure,  the  operator’s  initial  view  of  a  house  is  from  the 
camera  mounted  on  UAV  1 .  Next,  the  operator  views  a  semi-continuous  transition  implemented 
with  synthetic  vision/augmented  reality  technologies.  The  transition  consists  of  a  fly-out  portion 
(triangle  shapes  numbered  Points  1  and  2),  a  “bird’s  eye  view”  of  UAV  2  (Point  4),  and  a  fly-in 
portion  (Points  4  and  5).  The  “fly-out”  phase  of  the  transition  was  similar  in  structure  to  the 
“fly-in”  phase,  but  operated  in  reverse.  The  operator’s  final  view  of  the  tank  was  from  the 
camera  mounted  on  UAV  2.  The  entire  transition  lasted  6.5  seconds.  Further  detail  on  each 
segment  of  the  transition  is  provided  below  and  in  Figure  27.  The  timeline  in  Figure  27  shows 
the  duration  and  rate  of  change  (exponential  factor)  for  each  segment,  as  well  as  the  timing  of 
experimental  script  prompts  to  change  the  mission  (Circle  A)  and  locate  a  specific  tank  target 
(Circle  B). 

•  Point  1 ;  The  camera  view  switched  from  a  (simulated)  live  video  feed  to  a  purely 
synthetic  environment  from  a  VC.  The  imagery  from  the  VC  started  at  the  same  position 
with  which  the  live  video  ended:  12,000  to  24,000  ft  away  from  the  house  (varied 
because  the  UAV  is  loitering),  and  7,000  ft  AGL  giving  a  pitch  of  60  to  74  degrees  from 
horizontal.  From  this  start-point,  the  VC’s  FOV  opened  from  0.41  degrees  (horizontal)  to 
48  degrees  over  the  course  of  1 .25  seconds,  gradually  increasing  the  rate  at  which  it 
opened  by  an  exponential  factor  of  0.5.  This  gave  the  visual  impression  of  flying  away 
from  the  house  along  a  vector  from  the  house  to  the  first  UAV  at  an  increasing  speed. 

•  Point  1  to  Point  2:  The  VC  began  to  physically  (in  the  virtual  environment)  move  away 
from  the  house  along  the  same  vector  for  6,000  ft  over  the  course  of  0.25  seconds  with  an 
exponential  change  in  speed  of  -1.2  to  Point  2.  This  gave  the  impression  of  slowing 
down  slightly  at  the  end  of  the  fly-out.  Overall,  the  participants  had  the  impression  of  a 
smooth  continuous  fly-out  that  started  slowly,  sped  up  in  the  middle,  and  ended  slowly. 

•  Point  3:  The  view  switched  immediately  from  the  environment  surrounding  the  first 
UAV  (Point  2)  to  the  environment  surrounding  the  second  UAV  (Point  4).  In  this 
respect,  the  transition  was  semi-continuous. 

•  Point  4:  The  VC  started  18,000  ft  away  from  the  second  UAV  along  the  vector  from  the 
sensor  view’s  stare  point  through  the  UAV  with  a  FOV  of  48  degrees  and  a  pitch  of  60  to 
74  degrees  from  horizontal. 

•  Point  4  to  Point  5:  The  VC  moved  towards  the  second  UAV  along  the  same  vector  over 
3.0  seconds  with  the  speed  increasing  at  an  exponential  rate  of  1.3.  This  gave  the 
impression  of  almost  standing  still  at  the  top  of  the  fly-in  and  then  accelerating  towards 
the  tank. 

•  Point  5:  Once  the  VC  reached  the  second  UAV  the  FOV  narrowed  to  0.41  degrees  in  2.0 
seconds  at  an  exponential  change  in  rate  of  -2.5.  This  gave  the  impression  that  the  VC 
was  still  moving  towards  the  target,  slowing  down  as  it  approached  the  truck. 
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Figure  27.  Illustration  of  Transition  for  Changing  Camera  Views  in  Multi-UAV  Operations 


Method 


Experimental  Design 

Performance  was  examined  while  participants  completed  trials  both  with  and  without  the  GITZ 
transition  in  a  multi-UAV  control  environment.  Mission  type  was  also  manipulated  (see  Table 
1).  Each  scenario  involved  multiple  static  and  dynamic  missions,  and  participants  were  told  that 
performance  would  be  recorded  during  all  missions.  However,  the  data  analyses  focused  on  the 
second  static-to-dynamic  mission  transition  in  Scenario  Type  1  and  the  dynamic-to-dynamic 
mission  transition  in  Scenario  Type  2.  This  factor  was  included  to  determine  if  the  utility  of  the 
GITZ  tool  was  a  function  of  the  mission  types  involved  in  the  transition.  (Note;  for  trials  with 
the  GITZ  tool,  the  format  was  only  presented  when  transitioning  to  a  dynamic  mission,  from 
either  a  static  or  dynamic  mission.) 


Table  1,  Scenario  Types  Completed  by  Participants: 
Data  Analysis  Focused  on  Shaded  Cells 


Scenario 

Order  of  Four  Mission  Types 

Presented 

1 

Static 

Dynamic 

Static 

Dynamic 

2 

Static 

Dynamic 

Dynamic 

Static 

A  within-subjects  2x2x2  design  was  utilized  with  13  participants  (average  age  =  26.69  years) 
to  examine  the  GITZ  transition,  scenario/mission  transition  type,  and  replication  variables. 

There  were  a  total  of  eight  experimental  trials.  Trials  with  each  GITZ  transition  (off  or  on)  were 
blocked  with  participants  receiving  both  scenarios  (static-to-dynamic  or  dynamic-to-dynamic, 
two  replications  each)  with  one  GITZ  level  prior  to  receiving  the  other  GITZ  level.  Orders  of  the 
GITZ  on/off  level  and  mission  transition  type  were  counterbalanced  for  12  participants  and 
randomly  assigned  for  the  thirteenth.  Each  trial  lasted  approximately  12  minutes.  Total  session 
time,  per  participant,  was  approximately  4.5  hours  (including  2  hours  for  training). 
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Multi-UAV  Control  Simulation  Environment 


This  experiment  utilized  the  Vigilant  Spirit  multi-UAV  operator  eontrol  station  testbed  (Figure 
28).  The  testbed  eonsisted  of  two  24  ineh  monitors  (resolution  1920  x  1200  pixels),  a  keyboard, 
and  a  right  hand  joystick  and  mouse. 


Figure  28.  Vigilant  Spirit  Multi-UAV  Testbed 


The  left  monitor  (Figure  29)  presented  both  a  Global  Tactical  Situation  Display  (TSD)  (showing 
the  path  of  four  UAVs  performing  missions  over  different  urban  areas;  1290  x  1057  NM)  and  a 
local  TSD  (fixed  on  the  center  of  the  loiter  pattern  of  the  UAV  currently 


Left  Monitor 

A 


Global  TSD: 

Red  Plane  Task  Local  TSD 


Right  Monitor 


Current  UAV  All  UAV 
Camera  View  Camera  Views 


Comm  Task 


Chat  Windows 


iry  Info 


Health  and  Status 


Figure  29.  Illustration  of  each  Multi-UAV  Testbed  Window  Used  for  Experimental  Tasks 
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selected  (12.0  x  10.6  NM).  Figure  30  shows  a  sample  view  of  the  local  TSD  with  computer¬ 
generated  symbology  depicting  the  currently  selected  UAV,  mission  forces  information, 
sensor  footprint  (blue),  and  no-fly  zones  (red). 


Figure  30.  Sample  View  of  Local  (Selected  UAV)  Tactical  Situation  Display 

The  right  monitor  presented  the  view  from  the  gimbaled  camera  of  the  specific  UAV  selected 
with  an  overlay  of  Head-Up-Display  (HUD)  symbology  and  overlaid  synthetic  symbology, 
showing  enemy  (red)  and  friendly  (blue)  forces  and  the  Forward  Air  Control  (FAC)  center. 
Figure  31  shows  a  sample  (simulated)  camera  view  with  overlaid  symbology  depicting  red 
and  blue  mission  forces  information  and  the  FAC  (pink  cone).  During  each  GITZ  transition 
(between  Points  1  and  5,  Figure  27),  the  video  turned  monochrome,  the  crosshairs  became 
white,  and  the  HUD  symbology  overlay  was  blanked.  This  was  done  because  previous 
research  [18]  showed  the  importance  of  clearly  distinguishing  between  the  synthetic  and  real 
world  view.  The  video  imagery  was  depicted  with  the  MetaVR  sensor  visualization  software 
package  (VRSG^'^)  Version  5.3.  The  video  and  symbology  overlay  were  updated  at  10  Hz. 
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Figure  31.  Sample  of  (Simulated)  Camera  View 

To  the  right  of  the  eamera  window  were  four  UAV  thumbnails,  the  selection  of  which 
changed  other  windows  (local  TSD,  camera  view,  and  local  UAV  chat  window)  to  formats 
specific  to  that  UAV.  The  other  windows  on  the  two  monitors  (summary  panels  for  each 
UAV,  communications  task  matrix,  health/status  matrix,  and  other  chat  windows)  were  used 
for  secondary  mission-related  tasks.  Verbal  prompts  for  several  task  types  were  issued  over 
the  computer’s  speaker  system  and  participants  heard  an  audio  alert  whenever  the 
designation  button  on  the  joystick  was  pressed  over  the  correct  selection  area  defined  for  a 
target.  Besides  the  keyboard,  there  was  a  right  handed  mouse  and  joystick.  The  mouse  was 
used  during  static  missions  for  cursor  control  to  select  windows  and  complete  secondary- 
tasks.  The  joystick  was  used  in  dynamic  missions,  and  allowed  manipulation  in  two  degrees- 
of-freedom  for  control  of  camera  orientation  in  azimuth  and  elevation.  The  larger  center  “hat 
switch”  controlled  camera  zoom  and  the  center  upper  button  was  used  to  designate  locations 
indicated  by  the  crosshairs. 

Primary  Experimental  Tasks 

Each  trial/scenario  consisted  of  multiple  dynamic  (close  air  support)  and  static  (surveillance) 
missions  (Table  1).  Participants  received  multi-modal  commands  (verbal  prompt  and  chat 
message)  when  mission  transitions  should  occur;  this  information  identified  the  next  UAV 
and  mission  type. 

•  Dynamic  Missions;  participants  were  tasked  with  locating  and  designating  two 
enemy  tanks,  according  to  prompts  received  verbally  and  in  a  chat  window. 
Participants  were  also  tasked  with  remembering  the  tank  types  (no  barrel/short 
barrel/long  barrel,  the  tank  shape  (dome/box/cone),  and  color  (green/magenta/yellow) 
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of  the  symbol  denoting  the  FAC.  At  the  end  of  eaeh  dynamic  mission,  a  situation 
awareness  probe  regarding  the  tank  types  or  FAC  symbol  appeared  in  the  local  UAV 
chat  window,  and  participants  entered  their  response  in  the  same  window  via  the 
keyboard.  Two  examples  include;  “What  was  the  color  and  shape  of  the  FAC?”  and 
“What  type  of  tank  did  you  designate  first  in  this  mission?”  The  intent  of  the  probe 
was  to  detect  negative  transfer  of  context,  as  accuracy  would  depend  on  knowledge  of 
the  current  camera  view,  as  opposed  to  the  previous  camera  view.  Participants  had 
two  minutes  to  complete  each  dynamic  mission.  Response  time  and  accuracy  for  the 
tank  designation  and  probe  tasks  were  recorded,  as  well  as  the  efficiency  with  which 
the  camera  was  moved. 

•  Static  Missions;  participants  were  tasked  with  monitoring  the  video  feed  for  the 
selected  UAV  and  typing  “truck”  in  the  UAV  chat  window  when  a  truck  appeared  in 
the  video.  Each  time  a  truck  was  displayed,  it  remained  in  the  camera  view  for  only 
10-15  seconds.  The  percent  of  trucks  detected  was  recorded,  as  well  as  response 
time.  Static  missions  lasted  between  2-5  minutes.  During  static  missions, 
participants  monitored  the  camera  view  from  one  UAV  at  a  time,  although  in  some 
missions,  there  was  a  command  to  switch  to  a  different  UAV  and  monitor  its  camera 
view  for  the  appearance  of  a  truck.  During  static  missions,  the  camera  view  was 
automatically  zoomed  in  all  the  way  and  the  joystick  was  inactive  to  prevent 
participants  from  gaining  awareness  of  the  surrounding  area  that  might  influence 
performance  on  subsequent  missions.  When  the  camera  controls  were  inactive,  the 
crosshairs  changed  from  magenta  to  green,  as  an  additional  cue  of  camera  state. 


Secondary  Experimental  Tasks 

During  the  static  missions  of  each  scenario,  participants  were  required  to  complete  several 
types  of  secondary  tasks  (see  below).  During  the  dynamic  missions,  the  prompts  for  the 
secondary  tasks  were  still  presented.  However,  participants  were  trained  to  not  complete  the 
secondary  tasks  during  these  short  dynamic  missions  and  instead  focus  their  attention  on  the 
critical  tasks  involving  the  tanks  and  FAC.  This  procedure  was  used  as  it  is  anticipated  that 
in  operational  multi-UAV  dynamic  missions,  such  housekeeping  tasks  would  be  tabled  until 
after  the  critical  tasks  are  completed. 

There  were  five  types  of  secondary  tasks.  These  tasks  are  described  below  and  the 
corresponding  interfaces  are  illustrated  in  Figure  29.  These  tasks  were  designed  to  represent 
the  type  and  range  of  activities  anticipated  for  multi-UAV  supervisory  control.  Each  of  these 
secondary  tasks  occurred  frequently  during  the  scenarios  such  that  the  participants  received  a 
tasking  approximately  seven  times  each  minute. 

•  Switch  UAV/Camera  Eeed;  Participants  received  prompts  (verbal  and  chat  window) 
indicating  which  UAV  to  employ  for  the  next  mission.  In  response,  participants 
positioned  the  cursor  on  the  corresponding  UAV  thumbnail  and  selected  it  with  the 
mouse.  Selection  of  the  thumbnail  changed  the  Local  TSD,  camera  view,  and  chat 
window  under  the  camera  view  to  formats/views  appropriate  to  that  UAV.  During 
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static  missions,  this  task  was  used  repeatedly  so  that  the  partieipants  sequentially 
monitored  the  eamera  views  for  2-4  UAVs.  During  dynamic  missions,  participants 
were  only  requested  to  switeh  to  a  different  UAV  at  the  eompletion  of  the  mission. 
The  time  from  the  prompt  until  eorreet  thumbnail  seleetion  was  reeorded. 

•  Provide  Requested  Information:  Partieipants  responded  to  requests  for  information 
(prompted  via  one  of  the  ehat  windows)  by  typing  information  (UAV  altitude, 
airspeed  or  heading)  into  the  appropriate  ehat  window.  Data  from  this  task  were  not 
analyzed. 

•  Monitor  for  Unexpeeted  Airplane  Symbol:  Oeeasionally,  a  red  plane  symbol 
(representing  an  unexpeeted  new  hostile  threat)  appeared  on  the  Global  TSD  and 
remained  for  up  to  10  seeonds.  To  respond,  partieipants  eentered  the  eursor  on  the 
aireraft  and  seleeted  it  with  the  mouse.  The  symbol  disappeared  once  it  had  been 
designated  or  when  the  time  period  expired.  Measures  ineluded  number  of  symbols 
deteeted  and  time  from  the  symbol  appearance  until  its  designation. 

•  Monitor  Health  and  Status  of  Vehieles:  A  matrix  at  the  bottom  of  the  right  monitor 
showed  the  health/status  of  five  subsystems  (eolumns)  for  eaeh  of  the  four  UAVs 
(rows).  The  eells  were  green  when  the  health/status  was  nominal.  When  a  subsystem 
exceeded  a  pre-established  threshold  (seript  driven),  the  eorresponding  eell  turned 
yellow  on  the  matrix.  The  partieipants’  task  was  to  seleet  and  designate  the  yellow 
eell  with  the  mouse.  If  the  partieipant  failed  to  seleet  the  yellow  eell,  it  turned  red 
after  10  seeonds.  If  still  not  selected  by  the  partieipant,  the  eell  returned  to  green  after 
5  seeonds  and  a  time-out  was  reeorded.  Performanee  measures  ineluded  eompletion 
time  (time  from  eolor  ehange  until  eell  was  seleeted)  and  pereent  deteeted. 

•  Monitor  and  Respond  to  Audio  Communieations:  A  version  of  the  Coordinate 
Response  Measure  (CRM),  a  eommunieation  performanee  task  was  employed  [19]. 
Partieipants  continually  heard  a  string  of  eall  signs  and  color-number  eombinations, 
all  embedded  within  a  earrier  phrase.  For  example,  in  “ready  Eagle,  go  to  blue  five 
now,”  “Eagle”  is  the  call  sign,  and  “blue  five”  is  the  eolor-number  eombination, 
meaning  the  button  labeled  with  a  “blue  5”  should  be  seleeted  from  the 
eommunieations  matrix  window  at  the  bottom  of  the  left  monitor.  The  partieipant 
was  assigned  a  eall  sign  and  instrueted  to  make  the  appropriate  button  choiee  with  the 
mouse  every  time  that  eall  sign  was  issued.  Audio  eommands  were  issued  for  4  eall 
signs,  one  approximately  every  15  seeonds.  The  participants’  call  sign  was  issued 
approximately  once  every  2  minutes.  Performanee  measures  ineluded  eompletion 
time  (the  time  from  when  the  eall  sign  was  issued  until  the  eoordinate  button  was 
seleeted)  and  pereent  deteeted. 
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Procedures 


Participants  first  received  a  general  overview  of  the  experiment  (Appendix  E).  This  was 
followed  by  detailed  training  and  praetiee  trials  were  eompleted  until  performanee  stabilized. 
Next,  experimental  trials  were  eompleted.  Note  that  there  was  no  flight  eontrol  task  in  these 
trials;  eaeh  UAV  automatieally  loitered  in  a  eireular  pattern.  The  partieipants’  eontrol  inputs 
involved  moving  the  eursor  and  making  inputs  via  the  keyboard  for  the  statie  missions  and 
the  various  seeondary  tasks.  They  also  manually  eontrolled  the  eamera  viewpoint  in  the 
dynamie  missions,  via  the  joystiek,  to  loeate,  zoom  in  on,  and  designate  speeifie  targets. 

After  eaeh  experimental  trial,  rating  seales  were  administered  to  obtain  the  partieipants’ 
impression  of  that  partieular  trial  in  terms  of  situation  awareness,  task  diffieulty,  and 
workload.  Partieipants  also  eompleted  a  final  questionnaire  after  the  study  asking  them  to 
eompare  the  GITZ  and  no-GITZ  eonditions  and  provide  eomments  on  the  strategy  they  used 
for  task  eompletion.  (See  Appendix  F  for  all  questionnaires  used  in  this  evaluation.) 

Results 

Results  are  presented  for  eaeh  experimental  objeetive  raised  earlier:  in  a  multi-UAV  eontrol 
environment,  does  the  GITZ  transition  improve  partieipants’  overall  situation  awareness  and 
performanee  on  target  designation  tasks,  without  hurting  performanee  on  other  seeondary 
mission-related  tasks? 

GITZ  Transition  Impact  on  Overall  Situation  Awareness 

Participants’  ratings  on  the  Post-Trial  Questionnaire  indicated  they  had  more  situation 
awareness  in  trials  with  the  GITZ  transition,  compared  to  trials  without  the  GITZ  transition 
(F(l,12)  =  5.493, =  0.037).  (Ratings  did  not  significantly  differ  (all p  >  0.2)  for  the  other 
rating  scales  pertaining  to  task  difficult,  and  workload.)  On  the  post-experiment 
questionnaire  Final  Questionnaire,  nine  of  the  thirteen  participants  indicated  that  they  had 
more  situation  awareness  with  the  GITZ  transition.  The  other  four  participants  rated  their 
situation  awareness  as  equal  with  and  without  the  GITZ  transition. 

Another  measure  of  situation  awareness  was  the  participants’  performance  on  the  situation 
awareness  probe  administered  in  the  dynamic  mission.  Response  time  to  the  probe  did  not 
differ  significantly  as  a  function  of  whether  the  GITZ  transition  was  present  or  not  (F(l,12)  = 
1.522,  p  =  0.241).  Accuracy  was  also  similar  across  the  conditions;  five  and  four  errors  were 
made  with  and  without  the  transition,  respectively.  However,  collapsing  across  transition 
condition,  response  time  to  the  probe  was  faster  for  the  dynamic-to-dynamic  scenarios 
compared  with  static-to-dynamic  scenarios  (F(l,12)  =  8.798, />  =  0.012). 

GITZ  Transition  Impact  on  Target  Locate/Designate  Task 

To  address  this  objective,  the  metric  used  was  the  participants’  performance  in  locating  and 
designating  the  first  tank  after  transitioning  to  the  new  FlAV/camera  view  (in  the  second 
dynamic  mission  of  each  scenario).  First,  the  time  it  took  to  designate  the  tank  was  analyzed. 


39 


For  these  data,  the  results  showed  a  significant  difference.  Participants  took  about  six 
seconds  longer  to  locate/designate  the  tank  target  when  the  GITZ  transition  was  present 
compared  to  when  it  was  absent  (means=28.7  and  22.6  seconds,  respectively;  F(l,12)  = 
6.709,  p  =  0.024).  Since  the  response  time  period  analyzed  was  from  the  issue  of  the  prompt 
identifying  the  target  until  the  target  was  designated,  the  time  spent  presenting  the  transition 
format  was  not  a  factor  in  this  result. 

Further  examination  of  the  raw  data  indicated  that  this  time  difference  between  transition 
conditions  can  be  attributed  to  the  variability  in  initial  camera  zoom  state.  In  trials  without 
the  GITZ  transition,  participants  used  the  6.5  seconds  after  the  ‘switch  mission’  prompts  to 
zoom  the  camera  out.  With  the  camera  zoomed  out,  more  of  the  synthetic  symbology 
denoting  tanks,  FAC,  etc.  was  visible.  There  was  a  similar  response  in  the  trials  with  the 
GITZ  transition.  As  soon  as  participants  regained  control  of  the  joystick  after  the  6.5  second 
transition,  they  zoomed  the  camera  out  in  order  to  quickly  move  the  camera’s  viewpoint  to 
the  tank’s  location  (this  strategy  was  learned  from  reviewing  post-experiment  comments). 
Thus,  a  completely  zoomed  out  camera  state,  a  prerequisite  for  rapid  camera  viewpoint 
translation,  was  achieved  in  the  trials  without  the  GITZ  transition  prior  to  the  official  task 
start  time.  This  was  not  the  case  for  the  trials  with  the  GITZ  transition.  During  these  trials, 
part  of  the  recorded  task  completion  time  (over  5  seconds)  was  spent  zooming  out  the 
camera. 

To  account  for  the  effect  of  the  asymmetric  camera  state  on  task  completion  time,  the  data 
were  reanalyzed  using  a  modified  definition  of  task  time.  Task  time  was  measured  from 
when  maximum  camera  zoom  was  achieved,  in  both  conditions,  to  the  time  the  designation 
button  was  pressed.  This  time  period  more  accurately  reflects  the  time  each  participant  spent 
moving  the  camera  to  the  tank,  zooming  in  on  it,  and  designating  it.  Analyses  of  these  data 
indicated  that  the  average  time  to  locate/designate  the  tank  target  was  slightly  faster  (2.3 
seconds)  with  the  GITZ  transition  compared  to  when  the  transition  was  not  presented. 
However,  this  difference  was  not  significant  (F(l,12)  =  2.054, =  0.177). 

The  above  analyses  are  for  the  first  tank  designation  task  in  the  dynamic  missions,  as  it  was 
hypothesized  that  any  beneficial  effect  of  the  GITZ  transition  would  be  transitory  in  nature, 
diminishing  as  the  operator’s  interactions  with  the  new  environment  accumulated.  As 
expected,  data  analyses  for  the  task  to  locate  and  designate  the  second  tank  revealed  no 
significant  effects  as  a  function  of  transition  condition. 

Next,  the  efficiency  of  the  participants’  movement  of  the  camera  toward  the  tank  target  was 
examined.  One  measure  was  based  on  the  participants’  initial  movement  of  the  camera.  The 
angle  formed  between  the  vector  describing  the  camera’s  instantaneous  movement  direction 
after  the  command  prompt  and  the  vector  defined  by  the  direction  of  the  target  from  the 
initial  camera  position  was  computed  (after  filtering  for  random  movements  associated  with 
grasping  the  joystick).  Results  showed  that  participants’  initial  camera  movement  was  more 
accurate  (by  approximately  12  degrees)  when  the  GITZ  transition  was  present  (F(I,I2)  = 
5.969,  p  =  0.031)  compared  to  when  it  was  not  presented. 
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Another  camera  efficiency  measure  was  camera  path  length  (length  of  optimal  path  between 
camera  initial  position  and  target  location  divided  by  length  of  the  camera’s  actual  path). 

The  results  showed  a  significant  interaction  of  GITZ  transition  and  the  type  of  scenario 
(F(l,12)  =  5.024, =  0.045;  see  Figure  32).  Bonferroni  pair-wise  comparisons  of  the  data 
showed  that  with  the  GITZ  transition,  the  camera  was  moved  more  efficiently  during  static- 
to-dynamic  scenarios,  compared  to  dynamic-to-dynamic  scenarios  (t(12)  =  3.424, =  0.030). 
Without  the  GITZ  transition,  there  was  little  difference  in  average  path  length  (t(12)  =  1.018, 
;?=  1.00). 


Figure  32.  Average  Path  Length  Traversed  hy  the  Camera  as  a  function  of  GITZ  Transition 

Participants  were  asked  in  the  Final  Questionnaire  to  rate  the  extent  to  which  the  GITZ 
transition  impacted  performance  on  the  target  locate/designation  task.  Eleven  of  13 
participants  responded  that  GITZ  aided  performance  (the  other  2  participants  said  it  had  no 
impact). 

GITZ  Impact  on  Secondary  Tasks 

None  of  the  performance  measures  related  to  the  various  mission-related  secondary  tasks 
showed  significant  differences  as  a  function  of  transition  condition.  For  all  task  measures 
(response  time  to  switch  UAV  and  both  response  time  and  percent  detected  for  tasks  to 
monitor  and  respond  to  unexpected  aircraft,  health/status  alerts,  and  audio  communications), 
p  was  >0.1. 

The  subjective  data  supported  the  performance  data.  Ten  of  the  13  participants  responded  on 
the  Final  Questionnaire  that  the  GITZ  transition  had  no  impact  on  other  tasks.  (The  other 
three  thought  the  GITZ  transition  actually  aided  these  tasks.) 
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Discussion 


The  rationale  for  this  researeh  was  that  tailoring  a  transition  feature  (similar  to  what  ean  be 
downloaded  from  web-based  mapping  tools)  to  a  multi-UAV  eontrol  applieation  may  help 
operators  more  rapidly  dissoeiate  themselves  from  the  mission  environment  of  one  UAV  and 
quiekly  aequire  a  mental  map  of  the  new  mission  area.  Additionally,  having  an  expanded 
view  of  the  new  area  before  aequiring  the  new  eamera’s  viewpoint  may  provide  the  operator 
with  an  awareness  of  threats  and  terrain  features  outside  of  the  camera’s  current  FOV.  In 
sum,  it  was  thought  that  the  GITZ  transition  would  reduce  negative  transfer  effects  and 
enhance  situation  awareness,  resulting  in  improved  performance  on  the  tasks  required  in  the 
new  mission  (without  hindering  performance  on  other  mission-related  tasks). 

In  the  present  experiment,  the  subjective  data  indicated  increased  situation  awareness  when 
the  GITZ  transition  was  presented.  However,  the  objective  situation  awareness  probe 
performance  data  did  not  show  a  benefit  for  the  GITZ  transition.  Moreover,  a  significant 
reduction  in  time  to  locate  and  designate  the  target  after  switching  missions  was  not  realized 
with  the  transition.  The  GITZ  transition  did  slightly  aid  the  tank  target  designation  task  in 
terms  of  the  efficiency  of  camera  movement  (for  static-to-dynamic  scenarios  only)  and  the 
accuracy  of  the  initial  camera  movement.  Additionally,  the  GITZ  transition  did  not  hinder 
secondary  task  performance. 

This  was  the  first  evaluation  of  a  transition  aid  for  multi-UAV  control  applications.  Based  on 
the  fact  that  participants  rated  the  transition  format  favorably  in  the  questionnaires  and  some 
performance  measures  showed  a  benefit,  further  evaluation  of  this  transition  aid  for  multi- 
UAV  control  is  recommended.  However,  the  results  of  the  present  study,  combined  with 
comments  provided  by  participants,  indicate  that  several  issues  should  first  be  considered  in 
the  design  of  a  refined  transition  aid.  We  discuss  these  below. 

Transition  Design  Changes 

Participants  commented  that  the  transition  format  should  be  modified  in  three  aspects.  First, 
many  suggested  that  the  fly-out  portion  take  less  time  as  there  was  no  requirement  to  glean 
information  about  the  old  environment  when  switching  to  an  entirely  new  UAV/mission. 

With  this  change,  the  goal  of  the  shortened  fly-out  would  simply  be  to  cue  that  there  will  be  a 
new  view  and  that  it  is  time  to  dissociate  attention  from  the  current  UAV.  A  second 
recommendation  was  to  lengthen  the  time  spent  in  a  global  view  over  the  new  UAV  location. 
This  change  would  provide  more  time  to  retrieve  key  spatial  information  before  video 
movement  begins.  Third,  comments  suggested  that  the  speed  at  the  start  and  end  of  the  fly-in 
was  too  fast  for  effective  information  retrieval. 

Information  Retrieval/Control  Across  Displays 

In  this  experiment,  both  the  TSD  and  the  camera  view  included  overlay  symbology 
indicating  the  location  of  tanks,  etc.  However,  the  symbol  indicating  FAC  location  was 
intentionally  only  included  in  the  synthetic  symbology  on  the  camera  view.  During  dynamic 
missions,  the  participants  were  tasked  with  remembering  tank  and  FAC  description 
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information  in  order  to  answer  the  situation  awareness  probe.  During  trials  without  the  GITZ 
transition,  partieipants  foeused  their  attention  on  the  eamera  view,  looking  for  the  FAC 
symbol  in  addition  to  the  tank  they  were  to  designate.  In  eontrast,  many  participants  in  the 
trials  with  the  GITZ  transition  focused  their  attention  on  the  TSD  to  locate  the  commanded 
tank.  This  approach  was  likely  used  because  participants  were  able  to  learn  about  the  FAC 
and  general  spatial  relationships  from  the  synthetic  symbology  presented  on  the  camera  view 
during  the  transition.  Once  the  tank  target  was  in  the  sensor  footprint  on  the  TSD,  they 
switched  attention  to  the  camera  view  for  the  final  designation  steps.  However,  precise 
control  of  the  camera,  using  the  TSD,  was  difficult  as  the  sensor  footprint’s  movements  were 
dependent  on  its  orientation  relative  to  the  circular  orbiting  UAV.  Thus,  the  TSD-centric 
technique  employed  in  trials  with  the  GITZ  transition  may  have  lengthened  response  times 
for  tank  designations.  This  change  in  information  retrieval/control  strategy  between  the  trials 
with  and  without  the  transition  was  unexpected  and  follow-on  research  is  needed  to 
determine  what  specific  factors  play  a  role.  One  possibility  is  that  participants  had 
insufficient  training  with  the  transition  format.  Focused  demonstrations  may  have  been 
needed  on  the  ease  of  camera  control  using  the  camera  display  in  comparison  to  the  TSD. 
Experimental  manipulation  of  where  task-related  information  is  displayed  (camera  view 
versus  the  TSD)  would  also  be  informative.  Indeed,  there  are  numerous  cognitive  factors 
that  influence  information  retrieval,  task  switching  strategy,  and  performance,  such  as 
sensory  modalities,  response  modalities,  and  stimulus-response  mappings  [20]. 

Mission  Before  and  After  Transition 

Follow-on  research  should  further  examine  the  type  of  mission  scenarios  involved  in  the 
transition.  It  has  been  shown  that  task  switching  time  is  influenced  by  the  stimulus-response 
mapping  characteristics  of  the  two  tasks  [19].  This  finding,  together  with  the  results  from  the 
present  study  showing  performance  differences  as  a  function  of  mission  scenario  type, 
suggest  that  the  design  of  any  transition  aid  should  also  consider  the  specific  cognitive 
processes  involved. 

Operator  Control  of  Transition 

Given  that  many  factors  may  influence  the  method  of  task  switching  desired  for  a  particular 
situation  (e.g.,  mission  type,  relative  knowledge  of  the  area,  information  needs,  time 
available,  etc.),  there  may  not  be  a  ‘one  size  fits  all’  method  of  implementing  a  transition  for 
multi-UAV  applications.  It  may  be  preferable  for  the  operator  to  have  flexible  control  over 
the  transition  method  employed.  However,  the  advantages  of  this  flexibility  need  to  be 
weighed  against  any  concomitant  task  loading. 

Influence  of  Video  Imagery  Generator  Fidelity 

The  scene  generator  employed  for  this  study  did  not  have  the  resolution  required  to  permit 
the  use  of  easily  recognizable  landmarks  in  the  imagery  without  also  labeling  these  locations 
on  the  TSD  (and/or  via  overlaid  symbology  on  the  camera  image).  Having  a  system  that 
enables  display  of  high-resolution  ground  environments,  depicting  detailed  features,  would 
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be  more  representative  of  what  the  operator  views  in  operational  seenarios  (i.e.,  not 
everything  that’s  important  has  a  label)  and  may  enhanee  the  utility  of  a  transition  format. 
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SUMMARY 


This  first  instantiation  and  evaluation  of  a  transition  aid  for  multi-UAV  supervisory  eontrol 
was  very  informative  and  the  eoneept  shows  promise  for  benefiting  UAV  operator 
performanee  [21],  With  the  transition  format,  partieipants  rated  their  situation  awareness  as 
higher  and  there  were  improvements  in  eamera  movement  efficiency  measures.  However, 
this  transition  did  not  preclude  participants  from  needing  to  zoom  out  the  camera  view  after 
switching,  and  for  several  dynamic  mission  performance  metrics,  it  failed  to  provide  a 
significant  benefit. 

The  results  from  this  full  mission  simulation  evaluation  indicated  several  potential 
enhancements  which  may  increase  the  utility  of  the  GITZ  transition  for  switching  between 
UAV  camera  views.  First,  the  transition  format  needs  refinement,  ranging  from  the  speed  of 
various  transition  segments  to  whether  or  not  the  operator  has  direct  control  over  transition 
parameters.  The  findings  also  indicate  that  research  is  needed  to  determine  which  station 
display(s)  should  present  each  information  element  for  a  multi-UAV  control  application. 
Additionally,  this  experiment  showed  that  there  are  numerous  factors  that  may  influence  the 
utility  of  a  transition  aid,  including  the  nature  of  the  missions  involved  and  the  users’ 
strategy.  Finally,  this  experiment  illustrated  the  limitations  of  the  scene  generator  employed. 

Follow-on  research  is  underway  to  address  potential  enhancements  and  other  issues 
identified  as  a  result  of  this  evaluation.  It  is  also  planned  to  examine  how  best  to  implement 
a  transition  for  scenarios  that  involve  more  than  one  UAV  viewing  the  same  object/scene  for 
collaborative  operations.  The  use  of  transitions  for  improving  multi-UAV  operator  situation 
awareness  when  switching  to  a  different  UAV  and  its  associated  camera  view  is  just  one  of 
many  potential  benefits  of  applying  augmented  reality  technology  to  UAV  operations. 
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APPENDIX  A 


BACKGROUND  QUESTIONNAIRE 


Please  PRINT: 


Name:  _ 

Organization  (school  or  employment):  _ 

Address:  _ 

Phone:  _ 

Email:  _ 

Age  (yrs):  _  Gender:  Male  □  Female  □ 

Vision/Hearing  Normal  □  Corrected  to  Normal  □  Deficient  □ 

If  deficient,  please  describe;  _ 

Do  you  consider  yourself  especially  sensitive  to  motion  sickness  or  simulator  sickness? 
No  □  Yes  □ 

Experienced  with  3D  video  games  (games  with  a  “moving”  background)? 

No  □  Yes  □ 

If  yes,  estimate  how  much:  Seldom  □  Occasional  □  Frequent  □ 

Please  list  a  few  of  your  favorite  games,  or  games  you  have  the  most  experience  with: 

1.  _ 

2,  _ 

3.  _ 

Have  you  had  any  experience  with  sensor  imagery?  No  □  Yes  □ 

If  yes,  please  briefly  explain  the  extent  of  your  sensor  imagery  experience: 

Experienced  with  piloting  an  aircraft?  No  □  Yes  □  If  yes: 


IFR  Yes  No  Status?  Active  Duty  Guard  Reserve  Retired  Civilian 
Rated?  □  □  □  □  □  □  □ 

Total  Flying  Time:  Total  Jet  Time: 


Current  Aircraft  and  Hours: 

Other  Aircraft  and  Hours: 
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APPENDIX  B 


SIMULATOR  SICKNESS  QUESTIONNAIRE 

(at  start  and  end  of  each  experimental  section  &  if  symptoms  present) 

Please  cirele  one  severity  rating  for  eaeh  of  the  16  symptoms  listed  below.  Cirele  the  word  that 
best  matehes  your  eurrent  feeling  “right  now.” 


(0) 

(1) 

(2) 

(3) 

1. 

General  Discomfort 

None 

Slight 

Moderate 

Severe 

2. 

Fatigue 

None 

Slight 

Moderate 

Severe 

3. 

Headache 

None 

Slight 

Moderate 

Severe 

4. 

Eye  Strain 

None 

Slight 

Moderate 

Severe 

5. 

Difficulty  Focusing 

None 

Slight 

Moderate 

Severe 

6. 

Increased  Salivation 

None 

Slight 

Moderate 

Severe 

7. 

Sweating 

None 

Slight 

Moderate 

Severe 

8. 

Nausea 

None 

Slight 

Moderate 

Severe 

9. 

Difficulty  Concentrating 

None 

Slight 

Moderate 

Severe 

10. 

Fullness  of  Head 

None 

Slight 

Moderate 

Severe 

11. 

Blurred  Vision 

None 

Slight 

Moderate 

Severe 

12. 

Dizzy  (Eyes  Open) 

None 

Slight 

Moderate 

Severe 

13. 

Dizzy  (Eyes  Closed) 

None 

Slight 

Moderate 

Severe 

14. 

V ertigO  (loss  of  orientation  with 
respect  to  upright  (i.e.,  confusion 
“which  way  is  up”)) 

None 

Slight 

Moderate 

Severe 

15. 

Stomach  Awareness 

(discomfort  just  short  of  nausea) 

None 

Slight 

Moderate 

Severe 

16. 

Burping 

None 

Slight 

Moderate 

Severe 

If  you  have  additional  comments  on  the  above  symptoms,  please  include  them  helow. 


Are  there  any  other  symptoms  that  you  are  experiencing  right  now?  If  so,  please  describe 
the  symptom(s)  and  rate  their  severity. 

Time  □  Pre-Session  □  Post-Session  □  Other: 
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APPENDIX  C 


PILOT  STUDY  1  QUESTIONNAIRES 
POST-TRIAL  QUESTIONNAIRE 

1.  In  terms  of  your  ability  to  gain  situation  awareness  (SA)  of  the  entire  area,  compare  the  first  and  second 
transitions  ohserved.  SA  =  the  degree  to  which  you:  were  aware  of  important  elements  in  your  environment 
(i.e.,  landmarks,  targets,  major  features),  comprehended  your  current  location,  and  were  able  to  project 
future  status. 


Mark  1  of  the  3  boxes  below: 


SA  equal  with  Transition  1  and  2 

□ 

If  you  had  more  SA  with  one  of  the 
transitions,  how  much  more? 

Mark  1  of  the  3  columns  below. 

More  SA  with  Transition  1 

□ 

then: 

Slightly 

more 

> 

□ 

» 

□ 

Substantially 

more 

»> 

□ 

More  SA  with  Transition  2 

□ 

Comments: 


2.  In  terms  of  being  visually  appealing,  compare  the  first  and  second  transitions  observed. 

Mark  1  of  the  3  boxes  below:  If  you  found  one  transition  more  visually 


Transition  1  and  2  were  equally  visually 

appealing 

□ 

appealing,  how  much  more  appealing  was  it 
compared  to  the  other  transition? 

Mark  1  of  the  3  columns  below. 

Transition  1  more  visually  appealing 

□ 

then: 

Slightly  more 

> 

□ 

» 

□ 

Substantially 

more 

»> 

□ 

Transition  2  more  visually  appealing 

□ 

□ 

Comments: 


3.  In  terms  of  preference,  compare  the  two  transitions. 


Mark  1  of  the  3  boxes  below: 


Equal  preference  for 
Transition  1  and  Transition  2 

□ 

If  you  preferred  one  transition  more  than 
the  other,  how  much  more  did  you  prefer  it? 

Mark  1  of  the  3  columns  below. 

Prefer  Transition  1 

□ 

then: 

Slightly 

more 

> 

□ 

» 

□ 

Substantially 

more 

»> 

□ 

Prefer  Transition  2 

□ 

Comments  (If  you  preferred  one  transition  over  the  other,  please  discuss  why): 
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POST-BLOCK  QUESTIONNAIRE 


RELATIVE  JUDGMENTS  INSTRUCTIONS 

We  would  like  you  to  eompare  the  different  eoneepts  in  terms  of  personal  preference.  The  form 
for  doing  this  is  on  the  next  page  and  here  are  the  instruetions  for  filling  out  the  form; 

•  This  survey  consists  of  a  series  of  relative  judgments  comparing  two  concepts.  For  example,  Concept 
A  might  be  compared  to  Concept  B  in  terms  of  how  much  you  preferred  one  to  the  other.  A  nine- 
point  scale  will  be  used  for  comparing  the  preference  of  each  of  the  transition  types  or  transition 
times.  Examples  on  how  to  use  the  scale  are  given  next: 


If  you  prefer  neither  concept  “A”  nor  concept  “B”,  tl 

len  you  mark  the  form  in  the  center: 

<  «  «<  «« 

»»  »>  »  > 

Equal 

X  . 

"A" 

"B" 

If  you  prefer  “A”  a  little  more  than  “B”,  then  you  would  move  your  mark  a  little  closer  to  “A”: 


»»  »> 

» 

> 

Equal 

< 

« 

«<  «« 

"A" 

X  . 

"B" 

If  von  prefer”  A”  a  lot  more  than  “B”,  von  would  move  vour 

mark  very  close  to  “A”: 

«  «<  «« 

»» 

X  . 

»>  »  >  Equal  < 

"A" 

"B" 

On  the  other  hand,  if  von  prefer  “B”  moderatelv  more  than  “ 
in  that  direction: 

»»  »>  »  >  Equal  < 

A”,  then 

« 

X  . 

1  you  would  move  your  mark 

«<  «« 

"A" 

"B" 

Questions? 
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1.  Preference:  Please  compare  each  style  of  Fly-in  Concept  in  terms  of  which  yon  prefer. 


Fly-in  Concept 

»>  »>  » 

^  Equal 

< 

«  «<  «<  T-l  •  i-- 

Fly-in  Concept 

Long  Vertical  Drop 

-  -  - 

—  — 

— 

Short  Vertical  Drop 

Short  Vertical  Drop 

-  -  - 

—  — 

— 

Exponential  Sweep 

Exponential  Sweep 

-  -  - 

—  — 

— 

Long  Vertical  Drop 

Linear  Sweep 

-  -  - 

—  — 

— 

Exponential  Sweep 

Linear  Sweep 

-  -  - 

—  — 

— 

Long  Vertical  Drop 

Short  Vertical  Drop 

-  -  - 

—  — 

— 

Linear  Sweep 

Please  try  to  explain  yonr  above  ratings: 


What  one  fly-in  concept  do  yon  prefer  the  most  and  why? 


2.  Transition  time:  For  this  hlock  of  trials,  rate  how  you  felt  about  the  duration  of  time  during  the  transition. 


Transition  “Speed” 

Rating 

Too  Slow 

Good 

Too  Fast 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

52 


POST-SESSION  QUESTIONNAIRE 


1.  During  the  course  of  the  study  the  time  duration  of  the  different  fly-in  transitions  you  viewed  was 
varied.  Please  compare  your  preference  for  the  fly-in  duration  times  (2  seconds,  4  seconds,  or  6 
seconds). 


Preference:  Please  compare  each  “Fly-in”  time  in  terms  of  which  you  prefer. 


Fly-in  Time 

»>  »> 

» 

> 

Equal 

< 

«  «<  «< 

Fly-in  Time 

6  seconds 

-  - 

— 

— 

— 

— 

—  —  — 

2  seconds 

2  seconds 

-  - 

— 

— 

— 

— 

—  —  — 

4  seconds 

4  seconds 

-  - 

— 

— 

— 

— 

—  —  — 

6  seconds 

Please  try  to  explain  your  above  ratings: 

What  one  fly-in  time  do  you  prefer  the  most  (2  seconds,  4  seconds,  or  6  seconds)  and  why? 


2.  Preference:  Please  compare  each  style  of  Fly-in  Concept  in  terms  of  which  you  prefer. 


Fly-in  Concept 

»>  »>  » 

> 

Equal 

< 

«  «<  «< 

Fly-in  Concept 

Long  Vertical  Drop 

-  -  - 

— 

— 

— 

—  —  — 

Short  Vertical  Drop 

Short  Vertical  Drop 

-  -  - 

— 

— 

— 

—  —  — 

Exponential  Sweep 

Exponential  Sweep 

-  -  - 

— 

— 

— 

—  —  — 

Long  Vertical  Drop 

Linear  Sweep 

-  -  - 

— 

— 

— 

—  —  — 

Exponential  Sweep 

Linear  Sweep 

-  -  - 

— 

— 

— 

—  —  — 

Long  Vertical  Drop 

Short  Vertical  Drop 

-  -  - 

— 

— 

— 

—  —  — 

Linear  Sweep 
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Please  try  to  explain  your  above  ratings: 


What  one  fly-in  concept  do  you  prefer  the  most  and  why? 


3.  Was  there  a  strategy  you  employed  to  judge  if  one  transition  was  better  than  another? 


4.  Beyond  the  transition  concepts  you  saw  today  (transitioning  from  one  camera  viewpoint  to  another 
camera  viewpoint),  do  you  have  any  transition  ideas  that  you  feel  may  afford  more  awareness  of  an 
environment? 


5.  Do  you  have  any  comments  you  would  like  to  make  concerning  this  experiment  (e.g.,  briefing, 
training,  tasks,  etc.)? 


Thanks  for  your  participation. 
-The  SIRUS  Team. 
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APPENDIX  D 


PILOT  STUDY  2  QUESTIONNAIRES 
POST-TRIAL  QUESTIONNAIRE  -  PART  1 

1 )  For  this  trial,  draw  a  line  from  the  eenter  flag  to  the  direetion  of  the  (light  green,  light  blue, 
magenta,  yellow,  and  orange)  flag: 


Mark  the  box  (□)  which  best  reflects  your  ratings  for  each  of  the  following: 

2)  Situation  Awareness:  For  this  trial,  rate  your  overall  situation  awareness  —  the  degree 
to  whieh  you:  were  aware  of  important  elements  in  your  environment  (i.e.,  landmarks,  targets, 
major  features),  eomprehended  your  eurrent  loeation,  and  were  able  to  projeet  future  status. 


Low 

Situation  Awareness  Rating 

High 

Never 

Always 

Aware 

Aware 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

3)  Visually  Appealing:  For  this  trial,  rate  how  visually  appealins  the  fly-in  was. 


Visually  Appealing  Rating 

Lowest 

Appeal 

Highest 

Appeal 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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POST-PART  1  QUESTIONNAIRE 


For  Question  1  below,  we  would  like  you  to  compare  the  different  fly-in  concepts  in  terms  of 
personal  preference.  First,  here  are  the  instructions  for  filling  out  the  question: 

This  survey  consists  of  a  series  of  relative  judgments  comparing  two  concepts.  For  example,  Concept  A 
might  be  compared  to  Concept  B  in  terms  of  how  much  you  preferred  one  to  the  other.  A  nine-point  scale 
will  be  used  for  comparing  the  preference  of  each  of  the  fly-in  concepts.  Examples  on  how  to  use  the 
scale  are  given  next: 

If  you  prefer  neither  concept  “A”  nor  concept  “B”,  then  you  mark  the  form  in  the  center: 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X 

"B"  1 

If  you  prefer  "A"  a  little  more  than  "B",  then  you  would  move  your  mark  a  little  closer  to  "A": 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X 

"B"  1 

If  you  prefer  “A”  a  lot  more  than  “B”,  you  would  move  your  mark  very  close  to  “A”: 


»» 

X  . 

»>  »  >  Equal  <  «  «<  «« 

1  "A" 

"B"  1 

On  the  other  hand,  if  you  prefer  “B”  moderately  more  than  “A”,  then  move  your  mark  in  that 
direction: 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X 

"B"  1 

1.  Preference:  Please  compare  each  Fly-in  Concept  in  terms  of  which  you  prefer. 


Fly-in  Concept 

»>  »>  » 

Equal 

«  «<  «< 

Fly-in  Concept 

Linear 

—  —  — 

—  —  — 

—  —  — 

Shallow  Curve 

Shallow  Curve 

—  —  — 

—  —  — 

—  —  — 

Deep  Curve 

Deep  Curve 

—  —  — 

—  —  — 

—  —  — 

Linear 

Please  try  to  explain  your  above  ratings.  Also  identify  what  one  fly-in  concept  you  prefer 
the  most  and  explain  why.  (If  you  need  more  room,  use  the  baek  of  this  sheet.) 
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POST-TRIAL  QUESTIONNAIRE  -  PART  2 


1.  In  terms  of  your  ability  to  2ain  situation  awareness  (SA)  of  the  entire  area,  compare  the 
first  and  second  fly-in  observed, 

SA  =  the  degree  to  which  you:  were  aware  of  important  elements  in  your  environment  (i,e,, 
landmarks,  targets,  major  features),  comprehended  your  current  location,  and  were  able  to 
project  future  status. 


Mark  1  of  the  3  boxes  below:  If  you  had  more  SA  with  one  fly-in 


SA  eaual  with  Flv-in  1  and  2 

□ 

compared  to  the  other,  how  much 
more? 

Mark  1  of  the  3  columns  below. 

More  SA  with  Fly-in  1 

□ 

Slightly  more 

> 

Substantially 

More  SA  with  Fly-in  2 

□ 

then: 

» 

□ 

more 

»> 

□ 

□ 

Comments: 


2,  In  terms  of  being  visually  appealins-,  compare  the  first  and  second  fly-in  observed. 


Mark  1  of  the  3  boxes  below:  If  you  found  one  fly-in  more  visually 


Flv-in  1  and  2  were  eaually  visuallv 

appealing 

□ 

appealing,  how  much  more  appealing 
was  it  compared  to  the  other  fly-in? 

Mark  1  of  the  3  columns  below. 

Fly-in  1  more  visually  appealing 

□ 

Slightly  more 

> 

Substantially 

Fly-in  2  more  visually  appealing 

□ 

then: 

» 

n 

more 

»> 

□ 

□ 

□ 

Comments: 
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POST-PART  2  QUESTIONNAIRE 


For  Question  1  below,  we  would  like  you  to  eompare  the  different  fly-in  eoneepts  in  terms  of 
personal  preference.  First,  here  are  the  instruetions  for  filling  out  the  question: 

This  survey  consists  of  a  series  of  relative  judgments  comparing  two  concepts.  For  example.  Concept  A 
might  be  compared  to  Concept  B  in  terms  of  how  much  you  preferred  one  to  the  other.  A  nine-point  scale 
will  be  used  for  comparing  the  preference  of  each  of  the  fly-in  concepts.  Examples  on  how  to  use  the 
scale  are  given  next: 

If  you  prefer  neither  concept  “A”  nor  concept  “B”,  then  you  mark  the  form  in  the  center: 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X  . 

"B"  1 

If  you  prefer  “A”  a  little  more  than  “B”,  then  you  would  move  your  mark  a  little  closer  to  “A”: 


»»  »>  » 

> 

Equal  <  «  <«  «« 

1  "A" 

X 

"B"  1 

If  you  prefer  “A”  a  lot  more  than  “B”,  you  would  move  your  mark  very  close  to  “A”: 


»» 

X 

»>  »  >  Equal  <  «  «<  «« 

1  "A" 

"B"  1 

On  the  other  hand,  if  you  prefer  “B”  moderately  more  than  “A”,  then  move  your  mark  in  that 
direction: 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X 

"B"  1 

1.  Preference:  Please  compare  each  Fly-in  Concept  in  terms  of  which  you  prefer. 


Fly-in  Concept 

»>  »>  » 

^  Equal  ^ 

«  «<  «< 

Fly-in  Concept 

Linear 

—  —  — 

—  —  — 

—  —  — 

Shallow  Curve 

Shallow  Curve 

—  —  — 

—  —  — 

—  —  — 

Deep  Curve 

Deep  Curve 

—  —  — 

—  —  — 

—  —  — 

Linear 

Please  try  to  explain  your  above  ratings.  Also  identify  what  one  fly-in  concept  you  prefer 
the  most  and  explain  why.  (If  you  need  more  room,  use  the  back  of  this  sheet.) 
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FINAL  QUESTIONNAIRE 


For  Question  1  below,  we  would  like  you  to  eompare  the  different  fly-in  eoneepts  in  terms  of 
personal  preference.  First,  here  again  are  the  instruetions  for  filling  out  the  question: 

This  survey  consists  of  a  series  of  relative  judgments  comparing  two  concepts.  For  example.  Concept  A 
might  be  compared  to  Concept  B  in  terms  of  how  much  you  preferred  one  to  the  other.  A  nine-point  scale 
will  be  used  for  comparing  the  preference  of  each  of  the  fly-in  concepts.  Examples  on  how  to  use  the 
scale  are  given  next: 

If  you  prefer  neither  concept  “A”  nor  concept  “B”,  then  you  mark  the  form  in  the  center: 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X 

"B"  1 

If  you  prefer  “A”  a  little  more  than  “B”,  then  you  would  move  your  mark  a  little  closer  to  “A”: 


»»  »>  » 

> 

X 

Equal  <  «  <«  «« 

1  "A" 

"B"  1 

If  you  prefer  “A”  a  lot  more  than  “B”,  you  would  move  your  mark  very  close  to  “A”: 


»» 

X 

»>  »  >  Equal  <  «  «<  «« 

1  "A" 

"B"  1 

On  the  other  hand,  if  you  prefer  “B”  moderately  more  than  “A”,  then  move  your  mark  in  that 
direction: 


»» 

»> 

» 

> 

Equal 

< 

« 

«< 

«« 

1  "A" 

X  . 

"B"  1 

1.  Preference:  Please  compare  each  Fly-in  Concept  in  terms  of  which  you  prefer. 


Fly-in  Concept 

»>  »>  » 

Equal 

«  «<  «<  ^ 

Fly-in  Concept 

Linear 

—  —  — 

—  —  — 

_  _  _  Shallow  Curve 

Shallow  Curve 

—  —  — 

—  —  — 

_  _  _  Deep  Curve 

Deep  Curve 

—  —  — 

—  —  — 

_  _  _  Linear 

Please  try  to  explain  your  above  ratings.  Also  identify  what  one  fly-in  concept  you  prefer 
the  most  and  explain  why,  (If  you  need  more  room,  use  the  spaee  on  the  next  page.) 
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2.  Was  there  a  strategy  you  employed  to  judge  if  one  fly-in  concept  was  better  than  another? 


3.  Beyond  the  fly-in  concepts  you  saw  today,  do  you  have  any  fly-in  ideas  that  you  feel  may 
afford  more  awareness  of  an  environment? 


4,  Do  you  have  any  comments  you  would  like  to  make  concerning  this  experiment  (e.g., 
briefing,  training,  tasks,  etc.)? 


Thanks  for  your  participation.  ~The  SIRUS  Team. 
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APPENDIX  E 


FULL  MULTI-UAV  SIMULATION  EVALUATION:  SUBJECT  OVERVIEW 

Thank  you  for  participating  in  this  Synthetic  Interface  Researeh  for  UAV  (Unmanned 
Aerial  Vehiele)  Systems  (SIRUS)  Lab  experiment.  The  SIRUS  Lab  conduets  research  on 
candidate  UAV  operator  interfaees.  The  simulator  being  used  in  this  experiment  is  designed  to 
give  a  single  operator  control  of  multiple  UAVs.  This  experiment  will  test  a  new  UAV  to  UAV 
transition  display  ealled  Get  In  The  Zone  (GITZ).  GITZ  is  being  developed  as  a  tool  to  help 
operators  rapidly  gain  situation  awareness  of  a  new  area  when  switehing  camera  views  from  one 
UAV  to  another  and  to  mitigate  the  earry-over  effeets  eommonly  assoeiated  with  these  switches. 

Experimental  Scenario 

You  will  need  to  switeh  from  one  UAV  to  another  to  perform  surveillance  and  Close  Air 
Support  (CAS)  missions  while  communieating  with  different  eustomers  and  monitoring  UAV 
health  and  status,  air  traffic,  and  radio  chatter.  Speeifieally,  we  will  test  a  new  type  of  transition 
display  ealled  GITZ  to  determine  if  it  helps  you  perform  this  mission  better. 

In  this  experiment,  4  UAVs  are  flying  in  fixed  loiter  patterns  over  4  different  eities  under 
your  supervision.  You  will  be  tasked  with  switehing  among  these  UAVs  to  perform  surveillanee 
and  CAS  missions  while  you  perform  other,  more  seeondary  mission  tasks.  You  will  not  have 
control  over  the  UAVs’  flight;  you  will  only  be  able  to  switch  between  the  camera  view  of  one 
UAV  to  the  camera  view  of  a  different  UAV. 

Experiment  Simulator 

Vigilant  Spirit  Control  Station 


Global  TSD;  CmTeiitUAV  All  UAV 

Red  Plane  Task  Local  TSD  Camera  View  Camera  Views 
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Experimental  Tasks 

Upon  completion  of  training,  you  will  be  given  8  experimental  trials.  Eaeh  trial  will 
require  you  to  perform  surveillance  and  Close  Air  Support  (CAS)  missions  while  responding  to 
inquiries  and  monitoring  several  displays.  You  may  find  that  you  do  not  always  have  enough 
time  to  perform  all  tasks  so  it  is  essential  that  you  prioritize  your  tasks.  Some  tasks  are  more 
important  than  others.  For  the  purposes  of  this  experiment,  the  missions  are  the  primary  task  and 
have  priority  over  the  numerous  secondary  tasks. 

PRIMARY  TASK: 

Mission  During  the  trials  you  will  be  asked  via  both  radio  and  ehat  to  switch  UAVs  to  perform 
different  missions.  There  are  2  types  of  missions;  CAS  and  Surveillance.  In  each  trial  you  will 
be  asked  to  switch  back  and  forth  between  a  number  of  CAS  and  Surveillance  missions. 

1)  CAS  Missions  During  CAS  missions,  you  will  be  asked  to  loeate  and  designate  two  enemy 
tanks  in  an  area.  It  is  important  that  you  identify  and  remember  what  types  of  tanks  you 
designate.  There  are  three  different  types  of  tanks  (NB,  SB,  and  LB): 


Tank  1 :  No  Barrel  (NB)  Tank  2:  Short  Barrel  (SB)  Tank  3:  Long  Barrel  (LB) 


During  CAS  missions  you  must  also  identify  and  remember  the  shape  and  color  of  the 
Forward  Air  Control  center  (FAC).  There  are  three  different  shapes;  dome,  box,  and  cone  also 
three  different  colors;  green,  magenta,  and  yellow. 

At  the  end  of  each  CAS  mission  a  message  will  appear  in  the  UAV  chat  window  asking 
you  a  question  related  to  the  mission  (for  example:  what  was  the  shape  and  color  of  the  FAC, 
what  type  of  tank  did  you  designate  first  in  this  mission,  or  what  type  of  tank  did  you  designate 
second  in  this  mission).  You  must  respond  to  this  inquiry  as  quickly  and  accurately  as  possible 
by  typing  the  answer  in  the  chat  window  like  so: 

Q:  What  was  the  color  and  shape  of  the  FAC? 

A:  magenta  box  (magenta/green/yellow  |  box/dome/cone) 

Q:  What  type  of  tank  did  you  designate  first  in  this  mission? 

A:  NB  (NB,  SB,  LB) 

CAS  missions  require  your  full  attention  and  take  priority  over  everything.  Do  not 
attempt  to  perform  any  other  tasks  until  after  you  have  designated  the  second  enemy  tank  and 
answered  the  question  in  the  UAV  chat  window. 
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2)  Surveillance  Mission  During  Surveillance  missions  you  will  be  asked  to  monitor  the  video 
feed  for  a  pickup  truck.  It  only  appears  for  10-15  seconds  and  then  disappears  so  you  must  be 
vigilant.  The  truck  does  not  appear  in  every  surveillance  mission  and  may  appear  more  than 
once.  When  the  truck  appears,  report  it  by  typing  “truck”  in  the  UAV  chat  window. 

Workload  is  less  in  Surveillance  missions  compared  to  the  CAS  missions.  You  can  try  to 
continue  to  respond  to  inquiries  and  monitor  the  other  displays  while  performing  these  missions 
but  keep  in  mind  that  the  Surveillance  missions  are  a  primary  task  and  take  priority  over  other 
secondary  tasks. 

SECONDARY  TASKS 

1)  Chat  Throughout  the  trials,  questions  will  appear  in  the  4  chat  windows  at  the  bottom  of  the 
screen.  In  addition  to  the  CAS  Mission  questions,  there  will  be  questions  asking  you  for  your 
current  MSL  (altitude  relative  to  the  Mean  Sea  Level),  CAS  (Current  Airspeed),  and  HDG 
(Heading).  Locate  this  information  in  the  UAV  summary  info  panels  on  the  right  side  of  the 
station  and  answer  these  questions  as  quickly  as  you  can  by  typing  the  correct  number  in  the 
corresponding  chat  window.  The  monitoring  and  responding  to  the  chat  is  more  important  than 
the  other  secondary  tasks  because  CAS  Mission  questions  appear  in  chat.  Do  not  attempt  to 
answer  any  chat  questions  while  performing  CAS  Missions  with  the  exception  of  the  CAS 
Mission  questions. 

The  three  tasks  described  below  (Red  Plane,  Health  and  Status  and  Communications 
Monitoring)  are  all  of  equal  importance.  The  purpose  of  these  tasks  is  to  ensure  you  are 
scanning  both  screens  and  to  represent  the  workload  envisioned  for  operational  applications. 

2)  Red  Plane  A  red  plane  symbol  will  periodically  appear  and  disappear  on  the  global  Tactical 
Situation  Display  (TSD).  When  this  happens,  use  the  mouse  to  click  on  the  red  plane  before  it 
disappears.  This  simulates  monitoring  air  traffic  for  airborne  hostiles. 

3)  Health  and  Status  You  will  receive  warnings  throughout  the  trials  regarding  each  UAV’s 
systems  status.  Boxes  will  go  from  green  (normal)  to  yellow  (caution)  when  a  problem  occurs. 
Simply  click  on  any  yellow  box  to  return  the  health  and  status  to  normal.  If  you  fail  to  click  on  a 
yellow  box  quickly  enough  it  will  turn  red  (emergency)  indicating  that  the  problem  is  getting 
worse.  Click  on  any  red  box  to  return  the  system  to  normal. 

4)  Communications  Throughout  the  trials  you  will  hear  radio  chatter  that  will  consist  of 
phases,  each  beginning  with  a  call  sign  (e.g.  “Ready  Bravo  go  to  green  three  now,”  “Ready  Eagle 
go  to  blue  seven  now”).  You  will  be  assigned  the  call  sign  Eagle.  Whenever  you  hear  your  call 
sign,  click  on  the  color/number  button  the  radio  operator  asks  for  using  the  mouse.  Ignore  radio 
calls  that  are  not  addressed  to  you. 
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Again,  for  the  purposes  of  this  experiment,  prioritize  the  tasks  in  the  following  order  from 
highest  priority  to  lowest  priority: 


Highest  Priority 

Primary 

Tasks 

CAS  Missions:  Locate/Designate  Targets,  Post  Mission  Question 
Surveillance  Missions:  Find/Report  Targets 

Chat  Questions:  Report  UAV  Summary  Info 

1 

Secondary 

Health  and  Status:  Detect  Wamings/Emergencies 

V 

Tasks 

Communications  Tasks:  Detect/Follow  Radio  Instructions 

Lowest  Priority 

Red  Planes:  Detect/Select  Red  Planes 

Experiment  Requirements 

One  4-5  hour  session 

Training:  -'2  hours 

Experiment/Trials:  hours 

Questionnaires:  '-SO  min 

Approximately  12-16  subjeets 
20/20  vision,  normal  color  vision,  normal  hearing 
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APPENDIX  F 


FULL  MULTI-UAV  SIMULATION  EVALUATION  QUESTIONNAIRES 

POST-TRIAL  QUESTIONNAIRE 

Mark  the  box  (□)  which  best  reflects  your  ratings  for  each  of  the  following: 

1)  Situational  Awareness:  For  this  trial,  rate  your  overall  situation  awareness  —  the 

degree  to  whieh  you:  were  aware  of  important  elements  in  your  environment  (i.e.,  landmarks, 
targets,  major  features),  eomprehended  your  eurrent  loeation,  and  were  able  to  projeet  future 
status. 


Low 

Situation  Awareness  Rating 

High 

Never 

Always 

Aware 

Aware 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

2)  Perceived  Task  Difficulty:  For  this  trial,  rate  how  difficult  it  was  to  eomplete  your  tasks. 


Easy 

Task  Difficulty  Rating 

Hard 

Lowest 

Difficulty 

Highest 

Difficulty 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

3)  Perceived  Task  Performance:  For  this  trial,  rate  your  impression  of  how  well  you 
performed  your  tasks. 


Poor 

Performance  Rating 

Excellent 

Lowest 

Performance 

Highest 

Performance 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

4)  Workload  Estimate:  Provide  a  workload  rating  that  represents  your  workload  for  this 
trial.  For  ratings  greater  than  3  (Moderate  aetivity),  eomment  on  key  eontributors  to  your 
workload  for  that  trial. 


Workload  Rating 

Nothing 
to  do: 

No 

system 

demands 

Light 

Activity: 

Minimum 

demands 

Moderate 

Activity: 

Easily 
managed; 
Considerable 
spare  time 

Busy: 

Challenging  but 
manageable; 
Adequate  time 
available 

Very  Busy: 

Demanding  to 
manage; 
Barely  enough 
time 

Extremely  Busy: 

Very  difficult;  Non- 
essential  tasks 
postponed 

Overloaded: 

System 

unmanageable; 
Essential  tasks 
undone;  Unsafe 

1 

2 

3 

4 

5 

6 

7 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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FINAL  QUESTIONNAIRE 


Experimental  Conditions:  In  each  trial,  the  “Get-in-the-Zone”  (GITZ)  display  was  either 
‘on’  or  ‘off  during  transitions  between  camera  views  for  dynamic  missions: 

Without  a  Get-in-the-Zone  (GITZ)  display,  your  selection  of  a  different  UAV  for  a 
dynamic  mission  (target  search/designate)  abruptly  switched  the  tactical  map  of  interest 
and  camera  view  from  one  UAV  to  another.  The  displays  immediately  depicted  the 
different  terrain,  threat  environment,  and  camera  view  pertaining  to  the  newly  selected 
UAV. 

1)  Rate  helow  the  extent  to  which  transitioning  between  camera  viewpoints  without  a  GITZ 
display  was  distractinQ  to  you: 


Decree  Distracting 

Very  Distracting 

Slightly  Distracting 

Not  at  All 

-4 

-3 

-2 

-1 

0 

□ 

□ 

□ 

□ 

□ 

Comments: 


With  a  GITZ  display.  the  camera’s  view  transitioned,  retreating  upward  from  the 
starting  location  in  the  first  mission,  traveling  across  the  terrain  towards  the  new  location  at 
an  altitude  of  approximately  80,000  feet,  and  then  descending  to  the  view  of  the  camera 
located  on  the  newly  selected  UAV. 


2)  Rate  below  the  extent  to  which  transitioning  between  camera  viewpoints  with  a  GITZ 
display  was  distractinQ  to  you: 


Decree  Distracting 

Very  Distracting 

Slightly  Distracting 

Not  at  Ah 

-4 

-3 

-2 

-1 

0 

□ 

□ 

□ 

□ 

□ 

Comments: 
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3)  To  what  extent  did  the  GITZ  display  impact  performance  on  the  target 
search/desisnation  task  in  the  experimental  trials? 


Impact  (if  any)  on  Performance 

Hurt  Performance 

No  Impact 

Aided  Performance 

-2 

-1 

0 

1 

2 

□ 

□ 

□ 

□ 

□ 

Comments: 


4)  To  what  extent  did  the  GITZ  display  impact  performance  on  the  other  mission  related 
tasks  in  the  experimental  trials?  (UAV  switching,  responding  to  info  requests  and  SA  probes, 
deteeting  health/status  ehanges  and  unexpeeted  aireraft,  and  making  inputs  in  response  to  aural 
eommands) 


Impact  (if  any)  on  Performance 

Hurt  Performance 

No  Impact 

Aided  Performance 

-2 

-1 

0 

1 

2 

□ 

□ 

□ 

□ 

□ 

Comments: 


5)  Did  you  chanse  your  stratesv  ffor  any  of  the  tasks),  depending  on  whether  the  GITZ 
display  was  ‘on’  or  ‘off’? 

□  yes  □  no  If  yes,  please  explain  how  and  identify  for  which  task(s)  this  occurred. 


For  trials  in  which  the  GITZ  display  was  on,  GITZ  was  shown  in  two  types  of  mission 
transitions:  a)  Static  Mission  to  Dynamic  Mission  and  h)  Dynamic  Mission  to  Dynamic 
Mission.  Static  Missions  involved  monitoring  video  thumbnails  for  four  UAVs  and  Dynamie 
Missions  involved  loeating/designating  targets  with  a  single  UAV/eamera). 


6)  Did  the  usefulness  of  the  GITZ  display  depend  on  whether  you  were  transitioning  from  a 
static  mission  versus  transitioning  from  another  dynamic  mission? 

□  yes  □  no  If  yes,  please  explain. 
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8)  In  terms  of  your  ability  to  locate  and  desisnate  tarsets  in  the  dynamic  missions,  compare 
GITZ-On  with  GITZ-Off 

First,  mark  one  box  below:  If  you  had  better  task  completion  with 


Ability  equal  with  GITZ-On  &  -Off  □  then: 
Ability  better  with  GITZ-On  □ 

Ability  better  with  GITZ-Off  □  .J  — J 


If  you  had  better  task  completion  with  one 
of  the  GITZ  display  levels,  how  much 
more?  Mark  1  of  the  3  columns  below. 


Slightly  more 

» 

Substantially  more 

-► 

>□ 

□ 

»>  □ 

Comments: 


9)  In  terms  of  your  ability  to  accomplish  other  mission  tasks  (UAV  switching,  responding  to 
info  requests  and  SA  probes,  detecting  health/status  changes  and  unexpected  aircraft,  and 
making  inputs  in  response  to  aural  commands),  compare  GITZ-On  with  GITZ-Off 

First,  mark  one  box  below:  If  you  had  better  task  completion  with  one  of 


Ability  equal  with  GITZ-On  &  -Off 

□ 

then: 

the  GITZ  display  levels,  how  much  more? 
Mark  1  of  the  3  columns  below. 

Ability  better  with  GITZ-On 

□ 

Slightly  more 

» 

Substantially  more 

Ability  better  with  GITZ-Off 

□ 

>  □ 

□ 

»>  □ 

Comments: 
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10)  The  GITZ  transition  display  involved  three  segments: 

a)  retreating  upward  from  the  starting  location  in  the  original  (old)  mission 

h)  traveling  across  the  terrain  towards  the  new  UAV  location 

c)  descending  to  the  view  of  the  camera  located  on  the  UAV  of  the  new  mission 

Rate  below  the  aeeeptability  of  the  transition  speed  for  eaeh  of  the  three  segments: 


a)  Speed  of  Retreating  Upward  from  Camera  View  on  First  Mission 

Too  Slow 

Just  Right 

Too  Fast 

-2 

-1 

0 

1 

2 

□ 

□ 

□ 

□ 

□ 

b)  Speed  of  Traveling  A 

[cross  Terrain 

Too  Slow 

Just  Right 

Too  Fast 

-2 

-1 

0 

1 

2 

□ 

□ 

□ 

□ 

□ 

c)  Speed  of  Descending  to  Camera  View  for  New  Mission 

Too  Slow 

Just  Right 

Too  Fast 

-2 

-1 

0 

1 

2 

□ 

□ 

□ 

□ 

□ 

Comments: 


11)  Do  you  have  any  suggestions  on  how  the  implementation  of  the  GITZ  transition 
display  might  he  improved?  Or  other  aids  to  help  transition  from  one  camera  view  to 
another?  □  yes  □  no  If  yes,  please  explain. 


12)  Please  provide  any  additional  comments  concerning  the  experiment,  training,  tasks, 
and/or  simulator  you  might  have  (include  things  you  liked,  things  that  were  confusing, 
etc,): 


Thanks  for  your  participation!!  The  SIRUS  Team 
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LIST  OF  ABBREVIATIONS 


AFRL 

Air  Force  Research  Laboratory 

AGL 

above  ground  level 

ES 

exponential  sweep  (type  of  fly-in) 

FAC 

forward  air  controller 

ft 

feet 

FOV 

field-of-view 

GITZ 

Get  in  the  Zone  (Transition  Format) 

ISR 

intelligence,  surveillance,  and  reconnaissance 

LS 

linear  sweep  (type  of  fly-in) 

LVD 

long  vertical  drop  (type  of  fly-in) 

SA 

situation  awareness 

MSL 

mean  sea  level 

NM 

nautical  miles 

SVD 

short  vertical  drop  (type  of  fly-in) 

UAV 

unmanned  aerial  vehicle 

VC 

virtual  camera 
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